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Purpose 


Improving the quality of charter school authorizing is critical for improving the quality of 
charter schools and the outcomes experienced by students and communities. As every 
charter school has an authorizer, improvement in authorizing has the power to be among 
the most significant catalysts in creating many more high-quality schools and closing 
persistently failing schools that do not prepare students for future learning, civic 
participation, and workforce demands. 


Since 2009, the National Association of Charter School Authorizers (NACSA) has assessed 
the practices of the nation’s authorizers against its Principles & Standards for Quality 
Charter Schoo! Authorizing.1 As of 2015, NACSA had conducted more than 50 authorizer 
evaluations across 28 states, comprising authorizers overseeing more than 1 million 
students at the time of the evaluation. These evaluations were structured to be reflective 
and formative, allowing authorizers the opportunity to not only understand strengths in their 
authorizing enterprise, but also provide a clear understanding of practices in need of 
improvement. 


NACSA’s research and evaluation division conducted a thematic qualitative analysis of those 
evaluations. The analysis was designed to identify common strengths and weaknesses and 
ultimately to provide empirically based and detailed findings on specific practices 
authorizers are using across the country. It is anticipated that authorizers, NACSA, and other 
entities interested in improving authorizing will use these findings to design better technical 
assistance activities, professional development engagements, tools, templates, and other 
approaches that ultimately lead to a stronger sector of charter schools. 


1 http://www.qualitycharters.org/wp-content/uploads/2015/08/Principles-and-Standards_ 2015-Edition.pdf 
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About NACSA’s Authorizer Evaluations 


Consistent with NACSA’s Principles & Standards for Quality Charter School Authorizing, 
authorizer evaluations were organized to answer five guiding questions: 


e Application Decision Making: Did the authorizer approve applications based on 


applicants’ demonstrated preparation and capacity to open and operate a quality charter 


school? 


e Performance-Based Accountability: Did the authorizer have rigorous, appropriate 
standards by which it held schools accountable for results? Were decisions made with 
the intent to maintain high standards and protect the students’ and the public’s 
interests? 


e Performance Management Systems: Did the authorizer have effective systems for 
establishing and monitoring school performance expectations and holding schools 
accountable as necessary to protect students’ and the public’s interests? 


e Autonomy: Did schools have the autonomy to which they are entitled? 


e Organizational Capacity: To what extent did the organizational structure and systems 
support quality authorizing practices and forward the authorizer’s mission? 


Standards-based evaluations were completed by NACSA staff and external consultants 
selected and trained by NACSA. Funding for the majority of the evaluations came from the 


National Charter School Resource Center with support from the United States Department of 


Education’s Charter School Program. 


Evaluation teams gathered evidence through an extensive document review, Surveys, 
interviews, and a two-day site visit with authorizing staff, school leaders, board members, 
and other key stakeholders. Evidence-based ratings were provided for each primary and 
sub-domain, with a detailed written description justifying that rating. Findings from the 
evaluation were presented to key authorizing stakeholders, and those stakeholders were 
invited to provide fact-based clarification recommendations. The evaluation process 
concluded with a detailed written evaluation report that finalized ratings and summarized 
areas of strength and weakness for each of the standards. The entire evaluation process 
typically involved three evaluators. 


2 NACSA’s guiding questions and evaluation criteria have grown and evolved as we learned more about quality 
authorizing practices. Those guiding questions and standards underwent fairly significant revisions in 2011, 


but have been relatively stable since that time. The guiding questions listed in this document are those used in 


2015. 
3 Domains, sub-domains, and standards can be found in Appendix A. 
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Sample & Methods 


About the Sample 


The analysis used the final written evaluations from forty-six (46) evaluations completed 
between 2012 and 2015.4 Thirteen evaluations completed before 2012 were fully omitted, 
as they were not easily comparable to the current standards and the evaluation rubric. 
NACSA typically engaged authorizers with large numbers of schools in their portfolio, thus 
making the group of authorizer evaluations—and the sample used in this analysis— 
reasonably representative of large authorizers (i.e., those with 10 or more schools in their 
portfolio) but not representative of all authorizers across the country.® 


Summary of Methods Used 


The 29 sub-domain titles used by evaluators (see Appendix A) were used for the initial 
coding structure and did not change throughout the analysis process. Evaluations were 
uploaded to the online qualitative analysis software system Dedoose.® A team of three 
researchers coded written phrases within each sub-domain, placing each phrase in one of 
the 29 sub-domain codes and noting if the phrase was a strength or weakness as described 
by the text.’ The research team established strong inter-rater reliability by coding one 
evaluation and establishing agreement for 95% or more of the code categories and 
strength/weakness determinations. After establishing strong inter-rater reliability, 25 
percent of all subsequent coded evaluations were conducted by two researchers to ensure 
that inter-rater reliability remained high. 


The coding process resulted in 7,234 unique excerpts within the 29 sub-domains.® Four 
sub-domains were omitted from further analysis due to a low number of excerpts. Those four 
areas were in the Autonomy domain (Autonomy, Educational Program, Financial 
Management, and Differentiated Oversight); many autonomy-related areas were more easily 
described within other domains and they were included in those areas. The research team 
conducted a descriptive thematic analysis of excerpts within each sub-domain for commonly 
mentioned strengths and weaknesses. Other members of the research team and NACSA 
staff validated those themes.? 


4 Partial reviews were conducted for two authorizers prior to 2012 due to the number of schools in their 
portfolio. In each of those two instances, only data for outdated standards was omitted. 

> Please see Appendix B for data related to representativeness of authorizers in the sample. 

6 Two evaluations were difficult to upload properly and thus were manually read and coded. 

7 NACSA’s final written evaluations described strengths and weaknesses as “established” (i.e., what the 
authorizer had written and/or in policy) and “applied” (the authorizer’s activity in practice). Since the purpose 
of this analysis was to describe common strengths and weaknesses of authorizing and not necessarily 
differences in written vs. actualized practices, the research team coded across “established” and “applied” 
areas. 

8 The Organizational Capacity domain and its six sub-domains were added to the evaluation process in 2014. 
Thus, excerpts and findings from the Organizational Capacity domain are from 48 percent of the sample. 

9 Researchers followed the thematic analysis framework as described by Guest, G.S, MacQueen, K.M. & 
Namey, E.E. (2012). Applied thematic analysis. Thousand Oaks, CA: SAGE. 
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Findings and Report Structure 


The strengths and weaknesses described in this report are not intended to be exhaustive of 
all authorizing strengths and weaknesses. They represent the most salient strengths and 
weaknesses across the 46 authorizer evaluations as described by the evaluators of those 
authorizers against authorizing professional standards. Thus, this report presents a set of 
common, reliable, and important strengths and weaknesses among authorizer evaluations, 
but not all of the identified strengths and weaknesses apply to each individual authorizer in 
the field. In addition, noted strengths and weaknesses are evaluated against national 
professional standards and thus may not always be directly related to local student and 
public interest outcomes. 


What follows are the detailed findings from the thematic analysis process. Each of the four 
major sections (Application Decision Making, Performance-Based Accountability, 
Performance Management Systems, and Organizational Capacity) follows the same format 


e A brief description of what the domain seeks to assess 
e The sub-domains that comprise the primary domain 
e Asummary of key findings for that domain 


e Adetailed description of common strengths and weaknesses for each sub-domain 
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APPLICATION DECISION MAKING 


The Application Decision Making domain is organized to assess the degree to which the 
authorizer approves applications based on the applicant’s demonstrated preparation and 
capacity to open and operate a quality charter school. Eight sub-domains comprise this 
domain: 


e Application Materials and Processes 
e Educational Program 

e Organizational Plan 

e Business and Financial Plan 

e Capacity 

e Priorities and Application Adaptation 
e Decision Alignment 


e Transparency 


A number of strengths and weaknesses were found in each sub-domain, and those are 
described following a description of overall key findings on application decision making. 


Key Findings 


Alignment of Application Areas: Among the most consistently noted findings across sub- 
domains was the importance of evaluating applications for aligament—internal consistency 
and coherence—in addition to evaluating an application’s merits in discrete areas. Alignment 
was noted as both a critical strength and weakness across authorizers. Importantly, a key 
strength was that authorizers did not necessarily provide a preference for certain school 
missions or types of educational models, but rather that authorizers ensured all facets of 
the application were internally coherent and mutually supportive. One illustrative example 
was a physical arts-inspired school proposal. Evaluators noted that not only was the 
authorizer’s process strong in terms of evaluating the merits of the educational program and 
other domains, but also the degree to which all other parts of the application supported that 
school model. Not only did the authorizer evaluate the degree to which the budget 
supported each and every element of the educational program, but the authorizer also 
ensured that certain parts of the school’s facility supported those plans—in this case, 
making certain that the facility had space, permits, and safety pre-requisites to have a kiln in 
the school. Ensuring that all parts of the application were internally coherent and 
reinforcing—and not evaluating an application solely by its ability to meet standards in the 
discrete areas of education, business/finances, and organization—appeared to be a 
hallmark of strong authorizing; many authorizers were both strong and weak in this area. 


Requiring Application Elements But Not Evaluating Them: Requiring a number of 
application elements but not evaluating them was a significant weakness noted across 
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evaluations. Evaluators noted examples of application requirements but without any formal 
method of evaluating their adequacy via a rubric or other tool that described the authorizer’s 
expectations for those elements. Examples included requiring a full curriculum map that was 
not evaluated and an organizational chart, board development plans, 5-year cash flow plan, 
and descriptions of insurance plans without any criteria for judging adequacy. Stronger 
practices in this area included authorizers either (a) having criteria for judging the adequacy 
of these elements, or (b) requiring the applicant to sign a Statement of Assurances that 
certain elements were in place and met authorizer/state/federal legal requirements. 


Evaluation Conclusions, Justification, and “Feedback”: Providing applicants with a detailed 
justification for final decision making was noted as both an important strength and critical 
weakness. Rather than providing only a summary judgment (e.g., approve, not approve, 
approve with conditions) and/or judgments in specific domains (e.g., meets standard, 
partially meets standard), detailed, evidence-based reasons for those judgments were 
deemed a strength. The format of that justification varied from an aggregation of all 
evaluator comments into one document, to providing all evaluator comments individually to 
the applicant. Providing a justification, not just a summary judgment, not only enhanced the 
transparency of the decision-making process, but also provided the applicant with an 
opportunity to understand, and consider addressing, their deficiencies. 


The content of the justification was also considered critical. Evaluators noted that some 
authorizers used the evaluation justification as an opportunity to provide specific 
recommendations for improvement, while other authorizers explicitly did not provide 
recommendations for improvement, as they viewed it the responsibility of the applicant to 
figure out how to address deficiencies. This appeared to be a significant split in how 
authorizers viewed the role of application “feedback.” Evaluators did not specifically 
comment on the degree to which one approach was superior to the other but did note the 
risks inherent when an authorizer provides inappropriate feedback to an applicant (e.g., 
recommendations that explicitly told the applicant how to fix their application for approval), 
as it could minimize the opportunity for the authorizer to learn the degree to which an 
applicant has the capacity to address its own issues, an important capacity for a school 
during its charter term. 


Application Differentiation: Differentiating application requirements depending on the type 
of application received (e.g., start-up, replication), type of school proposed (e.g., virtual, 
Alternative Education Campus [AEC]) and who is involved in the application (e.g., charter 
management organization [CMO], education management organization [EMO], independent) 
was noted as a consistent strength and weakness. Having different application 
requirements, standards, and evaluation methods enabled authorizers to tailor their 
evaluation to the unique elements of the proposed school and provided an assessment that 
took into account specific application features. In some cases, application differentiation 
required applicants to provide different information than the standard application (i.e., an 
existing operator seeking to locate a school in a new state for the first time typically required 
a description of their current academic program rather than a theoretical description of their 
model), while other instances required applicants to provide significantly less information 
(i.e., a high-performing school/network in the same state typically required a description of 
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existing school academic, financial, and operational performance); in some cases, a 
replication petition could be very short with near-automatic approval. Unfortunately, not all 
authorizers differentiated and tailored their application materials and reviews to the type of 
application received, and not all conducted a thorough due diligence review of past operator 
performance. 


The remainder of the Application Decision Making domain describes specific strengths and 
weaknesses emerging from the thematic analysis for each sub-domain. It is of note that 
some practices could be considered both a strength and weakness. In those instances, it is 
likely that some authorizers were quite strong in those practices while others were quite 
weak. 


Application Materials and Processes 


Standard: The authorizer provides clear guidance and requirements regarding application 
materials and submission requirements and runs a clear and well-structured application 
process with realistic timelines. 


Key Strengths 
Evaluators provided a number of consistently cited strengths, including the following: 


Clear, measurable mission, with all components aligned and supporting that mission. 
While evaluators noted a number of components of application requirements, the most 
consistent strength was application materials and processes that required a clear and 
measurable mission, which thus ensured alignment of all components with that mission. 
Requiring applicants to describe their plans for academics, financial management, 
organizational management, facilities, working with educational service providers, and other 
components was routinely noted as a strength. Ensuring that application materials and 
systems specifically required how those functions supported the school’s mission was a 
clear differentiating strength. For example, if an applicant proposed a STEM (science, 
technology, engineering, and math) school serving underserved urban students as its 
mission, application evaluators were expected to assess not only the adequacy of the 
school’s proposed academic, financial, and facility plans (among others) but were also 
asked to evaluate the degree to which those plans supported a STEM school serving 
underserved students. 


Use of specific application processes. Evaluators consistently noted that strong application 
processes included the following practices: leading and/or participating in application 
information sessions; establishing application timelines and strictly adhering to those 
timelines; conducting application interviews with scripted questions drawn from a review of 
the application; comprehensive due diligence work, specifically conducting site visits for 
applicants currently operating one or more schools, and doing a comprehensive 
investigation of academic, financial, and operational reviews of existing schools and the 
operator; conducting community forums/public meetings on applicants, even if not required. 
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This is not to say that other application processes were not also useful and valuable, but 
evaluators noted these practices consistently as strengths. 


Evaluation rubric. Not only did strong application processes use a publicly available 
evaluation rubric, but the strongest justified the rating and/or decision by including detailed 
notes aligned with rubric criteria, which were then given to the applicant. Consistently noted 
practices included making the evaluation rubric available on the authorizer’s website and/or 
as part of the application packet explaining the content, format, and expectations of 
submitting an application. Further, providing notes and ratings from reviewers (either 
individually or aggregated) that provided clear reasoning for application acceptance or 
rejection was noted as a consistent strength. 


Training and frequent norming of all reviewers and evaluators. A consistently noted 
strength was providing training to any person formally reviewing the charter school 
application. Additionally, evaluators noted that frequently—typically once every two or three 
years if evaluators remained the same, and always for new evaluators—a norming process 
took place that ensured assessment teams scored the same application in similar ways. The 
goal of this norming process was to promote consistency in scoring across all evaluators 
(e.g., board members, authorizer staff, external reviewers). 


Key Weaknesses 


The most frequently mentioned weakness in application materials and processes related to 
how applications were reviewed: 


e No training was provided to reviewers, allowing them to rely solely on “professional 
expertise” in evaluating applications. 


e External reviewers were not included in evaluating applications. 


e Detailed comments/justification for application acceptance or rejection were not 
included; if comments were included, they were not aligned with the rubric criteria. 


e There were widely inconsistent ratings and findings among reviewers with no 
notes/comments explaining the reasons for divergent conclusions. 


Another consistently cited weakness was difficulty evaluating existing operators. Evaluators 
noted that some authorizers conducted little due diligence. They also noted that when due 
diligence was conducted, it often suffered from (a) not having a rubric and/or evaluation 
criteria for site visits to existing schools, and (b) asking operators for information that the 
authorizer did not have established criteria for evaluating and/or did not formally review or 
evaluate as part of the application process. 


How authorizers used evaluation rubrics was another consistently cited weakness. Not 
aligning the rubric to the authorizer’s performance framework and/or application criteria, 
having a rubric that did not cover all documents asked of the applicant, and having rubrics 
that were underdeveloped were frequently noted weaknesses. In addition, not making the 
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evaluation rubric available publicly and providing only verbal, not written, evaluation 
summaries to applicants were also noted as weaknesses. 


How information from public hearings was used also appeared to be a challenge. Evaluators 
noted that authorizers used a number of practices when conducting these hearings, 
including live streaming, providing data on who spoke in favor of or against the proposal, 
and any substantive comments made about the application itself (which was noted is very 
rare). How that data was used in actual decision making, however, was very unclear. 


The content of feedback that was provided by authorizing staff to applicants during the 
evaluation process was also cited as a weakness. It appeared this practice was used for 
applicants that were “close” to meeting the authorizer’s standard but failed to do so. 
Evaluators noted the inherent risk in an authorizer helping an applicant in this manner; 
rather than providing the applicant with a detailed evaluation of their proposal and allowing 
the applicant to address any deficiencies, authorizers ran the risk of not understanding the 
capacity of the applicant to address challenges. 


Educational Program 


Standard: The authorizer has thorough requirements and rigorous evaluation criteria for the 
proposed educational program, including vision and mission statements, educational 
philosophy, curriculum and instruction, teaching skills and experience, calendar and daily 
schedule, target population, enrollment, and plans for educating students with special 
needs. 


Key Strengths 


Data and information required in a school’s educational program were extensive and tended 
to include some areas beyond the areas listed in the standard. The most frequently 
mentioned items beyond those listed in the standard were (a) the evidence and/or theory of 
why the applicant’s educational program would work with the targeted student population, 
and (b) clear, measurable goals the educational program would achieve. Some authorizers 
also required such things as an applicant’s promotion policies, disciplinary policies, 
professional development plans for staff, and specific educational goals the applicant 
proposed to achieve. 


The most consistently noted strength in evaluating applicant educational programs was the 
clarity, alignment, and rigor of the authorizer’s evaluation rubric. Strong rubrics evaluated 
the degree to which the educational program elements aligned with the school’s mission, 
clear qualitative definitions for rating each element of the school’s educational program, and 
complete alignment between educational program application requirements and review 
criteria. 


In addition, evaluators noted that strong evaluation rubrics were evaluated rigorously for 


internal consistency, but did not provide for a preference in types of educational programs 
proposed. Stated differently, educational program evaluation criteria were aligned with the 
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school’s mission and other domains in the application, but allowed for wide diversity in the 
types of educational programs that could be proposed. 


Key Weaknesses 


The most consistently cited weakness was a lack of measurable academic outcomes or how 
those outcomes would be used if a charter were granted. Evaluators noted that many 
educational program goals were either not measurable and/or were process- rather than 
outcome-focused. In addition, goals proposed by the school were not used at all in the 
charter agreement and were never reviewed as part of the ongoing monitoring or 
accountability process. 


Problems related to how the evaluation rubrics were used and the kind of information they 
provided were also consistently noted weaknesses. Rather than providing a thorough 
description of strengths and weaknesses in evaluating the application, evaluation 
descriptions were often composed of a list of recommendations for improvement. Some 
authorizers used the evaluation rubric and process as a way of providing inappropriate 
technical assistance to applicants on how the applicant’s plan needed to be changed to 
meet the authorizer’s standards. 


Evaluators also noted that some applications required responses and supporting documents 
that were (a) not evaluated, and (b) not helpful or needed for evaluating the school’s 
educational plan (e.g., a curriculum map, learning standards that are often a re-statement of 
state academic standards). 


A final weakness was specific to the lack of evaluating alignment between the educational 
program and other areas of the application, especially the applicant’s budget. Evaluators 
noted that rubrics and review processes evaluated the school’s educational program in 
isolation rather than the degree to which it aligned with the applicant’s mission and ensured 
that all promises made in the educational program were adequately reflected in the 
applicant’s budget. 


Organizational Plan 


Standard: The authorizer has thorough requirements and rigorous evaluation criteria for the 
proposed organizational plan, including effective governance and management structures 
and systems (including staffing); the demonstration by founding team members of diverse 
and necessary capabilities; and the understanding of legal requirements related to opening 
and operating a charter school. 


Key Strengths 


Applications appeared to require a wide range of organizational elements. Some of the more 
frequently mentioned areas included recruitment/staffing/human resource plans; legal 
documentation (e.g., articles of incorporation, bylaws, board policies, background checks); 
plans for community support and engagement (and especially evidence of actual 
engagement prior to submitting the application); an organizational chart and key leadership 
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roles and functions; transportation plans; food service plans; board development plans, 
including how to ensure diverse experiences needed to oversee a school, as well as 
committee structure (if any); recruitment and admission policies; background and 
experience of the school leader (or detailed plans, criteria, and qualities if not yet hired); and 
facility plans, among others. 


The most consistently noted strength was that application organizational plans were 
required to be aligned with all other domains of the application, including areas that may not 
have appeared to require alignment. For example, rubrics and review processes that 
required the organizational plan to align with the school’s mission, financial plan, and 
educational plan were noted as clear strengths. One example provided was the degree to 
which an art-focused educational plan aligned with the requisite facility needs (e.g., space 
and safety requirements for a kiln in the school’s building). 


Evaluators also noted many strengths related to applicant organizational plans when 
working with an educational service provider (ESP). Applications that require (a) scope of 
service, (b) evidence of capacity, (c) a thorough board monitoring and accountability plan, (d) 
a copy of the actual (or strong initial draft) of the management agreement, (e) description of 
the relationship between the board and ESP (e.g., how the relationship commenced, how the 
board views its role in relation to the ESP), and a thorough description of services the board 
is purchasing from the ESP—and at what cost—were among the most frequently cited areas 
of strength among boards contracting with ESPs. 


Key Weaknesses 


Two clear weaknesses emerged from the analysis: evaluation of the organizational plan that 
was akin to a “checklist” and asking for organizational plan elements that were not 
evaluated. Often, rather than an evaluation that focused on quality, coherence, and 
alignment, both in substance and practice, the evaluation process for organizational 
elements were examined for their presence or absence. For example, rather than evaluating 
the degree to which the applicant’s facility aligned with their educational plan, rubrics 
required evaluators to check if the facility was secured, viable, and met health and safety 
requirements. In addition, evaluators noted occasions where the application required 
elements such as an organizational chart or board development plans, but without a 
corresponding evaluation rubric and set of criteria by which such plans would be judged. 


Evaluators noted a number of commonly missing organizational plan elements (é.g., 
transportation, food service). The most frequently mentioned missing element was a 
comprehensive start-up plan from application approval to school opening. 


Evaluators also noted a number of weaknesses related to applications seeking to work with 
an ESP. The lack of depth of due diligence was noted as a weakness, including an analysis 
of the viability of the ESP itself (e.g., its organizational health, sustainability, governance 
history). Further, not having criteria to evaluate the quality of due diligence work on ESPs 
was noted as a weakness on multiple occasions. 
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Business and Financial Plan 


Standard: The authorizer has thorough requirements and rigorous evaluation criteria for the 
proposed business plan, including financial viability of the plan demonstrated through 
budget projections that are aligned with the proposed educational program. 


Key Strengths 


The clearest strength emerging from the analysis was less about what authorizers required 
in an applicant’s business and financial plan but rather how authorizers evaluated the 
adequacy of that plan. Frequently mentioned examples include the following: 


Evaluating for alignment. Rather than examine the business and financial plan 
independently, authorizers who specifically examined alignment with other parts of 
the plan were noted as a strength. Examining alignment of the educational program 
with the business and financial plan appeared to be particularly important in order to 
ensure that all facets of the school’s educational plan were adequately accounted for 
in the applicant’s budget. A strong budget narrative was noted as particularly 
important to assess alignment. 


Scrutinizing enrollment projections. Practices related to heavy scrutiny of the 
number of students a school sought to enroll in its first year and beyond were noted 
as critical strengths. That included the authorizer (a) using their knowledge of the 
local community and past charter school enrollment data, (b) critically evaluating a 
school’s marketing and student recruitment plans, and (c) requiring a detailed 
contingency budget and plan (e.g., how the first-year budget and educational 
program would change if the applicant enrolled only % of their intended enrollment). 


Allowing only secured fundraising in the budget. Rather than allowing fundraising 
projections in an applicant’s budget, evaluators noted that strong application 
practices allowed for only secured fundraising (e.g., a letter indicating such or actual 
funds received) in the applicant’s budget. Some evaluators noted that authorizers 
might allow for non-secured fundraising to be included in an applicant’s budget, but 
the applicant would clearly have to identify those unsecured funds and what they 
would pay for, and justify how removing those unsecured funds from the budget 
would not adversely impact the applicant’s base educational program or operational 
plan (e.g., facility). 


Differentiated business/financial plan when working with an educational service 
provider (ESP). More business and financial plan information was typically required if 
an applicant proposed to work with an ESP. Evaluators noted strengths related to the 
authorizer asking for data related to prior schools (e.g., audits, financial records, 
governance arrangements) and the network itself, if the ESP operates as a network 
with “central” staff and supports. 


Authorizer-produced budget template. Greater consistency in evaluating an 
applicant’s budget could occur if the authorizer created and required its own budget 
template and narrative requirements. Evaluators noted that having a single budget 
template all new applicants were required to use could reduce misunderstandings of 


@ N a Cc S a Qualitative Analysis of Authorizer Evaluations 14 


the applicant’s plans and avoid misinterpretations of what the applicant was 
proposing. 


e Reviewers evaluating the entire proposal. Evaluators noted particular strengths in 
requiring reviewers—external reviewers, in particular—to read and evaluate the entire 
application instead of focusing merely on the business/financial section. While the 
scope of the review could be more limited, ensuring alignment between the 
business/financial plan and other areas could be sufficiently done only if the 
reviewers examined the entire application for alignment, rather than assessing the 
business/financial plan independent of the rest of the application. 


Key Weaknesses 


Vague criteria by which applicant business/financial plans were evaluated was the most 
consistent weakness noted. Evaluation rubrics and processes that provided only a 
summative rating (e.g., “sound,” “adequate,” “reasonable,” “unacceptable”) without a 
detailed, evidence-based rubric and comprehensive description underlying those ratings 
provided no justification for authorizer decision making. 


” 6 ” 


A second weakness was asking for business/financial plan information that was not 
evaluated nor used in decision making. For example, many authorizers asked for cash-flow 
data and insurance descriptions, but did not have a rubric for evaluating the adequacy of 
those materials nor their alignment with other parts of the application. 


A final weakness was related to the extent of due diligence conducted by authorizers for 
applicants seeking to work with an ESP. Evaluators noted wide differences in the extent to 
which authorizers asked for key financial data for ESP-managed schools and for the ESP 
itself. Areas not frequently asked for included budget-to-actual data for ESP-managed 
schools; enrollment variance (e.g., what was the target versus actual enrollment for each 
school); how governance had changed at each school (e.g., are board members fairly stable 
or has there been lots of turnover in the ESP’s network); and data on governance and 
finance for the ESP’s central capacity. 


Capacity 


Standard: The authorizer has thorough requirements and rigorous criteria for evaluating the 
applicant’s capacity to implement the school plan effectively, including but not limited to a 
substantive in-person capacity interview with all qualified applicants. 


Key Strengths 


Authorizers most typically evaluated the capacity of: (a) the proposed governing board, (b) 
the applicant team, (c) the school leader and leadership team, and (d) the educational 
service provider, if any. Evaluators indicated that authorizers emphasized evaluation the 
most (i.e., had the most evaluative criteria and processes) for the proposed governing board 
and its relationship with the school leader and/or ESP. 
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Conducting a detailed in-person capacity interview—and particularly how that interview was 
conducted—was noted as critical for effectively evaluating the capacity of the application. 
Evaluators noted that strong capacity interviews share the following characteristics: 


e The authorizer specified who on the applicant team must attend the capacity 
interview. 

e The interview was an adequate length to get through all major questions. 

e Targeted and scripted questions emanating from the application and review were 
developed; the most critical questions were prioritized. 

e The proposed governing board was evaluated for the requisite qualifications and 
experiences in overseeing the educational enterprise, but also the degree to which 
governing board members could describe how those qualifications and experiences 
would translate into effective governance. 

e Interview questions were aligned with the application criteria and evaluation rubric. 

e There was a method for accurately capturing answers to questions. 

e Time was scheduled immediately after the interview for the evaluation team to 
debrief and discuss responses received from applicants. 


Additional practices were noted as strengths if the applicant proposed to work with an ESP. 
Specifically, in addition to academic, financial, and operational due diligence done by the 
authorizer, authorizers required a description of how the board and ESP proposed to work 
together, and a description of how the collective capacity would drive results for the 
proposed student population. Further, having evaluation criteria that assessed the degree to 
which the governing board had effectively conducted a capacity evaluation of the ESP’s 
governance and leadership was seen as a strength. Evaluators also noted that an in-depth 
capacity interview with the ESP—typically in addition to the in-person interview and inclusive 
of a site visit to a school currently being managed by the operator—was a strength. 


Finally, heavy scrutiny of the school leader—and/or the board/ESP process for selecting the 
school leader—was noted as important. It was preferred if the school leader was identified in 
the application (with his/her qualifications included with the application materials) and was 
part of the capacity interview process. If the school leader had not been identified, 
evaluators noted the importance of the authorizer having evaluation criteria related to the 
process the board/ESP will go through to identify the school leader (e.g., criteria and 
experience aligned with the school’s mission and other elements of the application, 
outreach process, selection process, and accountability structure). 


Key Weaknesses 


The clearest and most consistent weaknesses that emerged from the analysis were 
authorizers not conducting an in-person capacity interview (or making it optional) and not 
adequately evaluating the capacity of ESPs. 


When in-person interviews were conducted, the following implementation weaknesses were 
noted: 
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e A lack of alignment between interview questions and evaluation criteria (e.g., using 
the majority of the interview to “get to know” the applicant team) 

e Little structure associated with the interview (e.g., free-flowing conversation rather 
than a purposeful attempt to have critical questions answered) 

e Nospecifics on who must attend the capacity interview 

e Nosystemic collection or documentation of applicant responses to key questions 
during the interview 


Further, evaluators noted strengths in due diligence processes (as described in prior 
sections), but also noted weaknesses related to evaluating the capacity of the ESP, by the 
authorizer directly and the applicant governing board. Evaluators noted weaknesses related 
to (a) authorizers not having criteria in place to evaluate the extent to which the applicant’s 
governing board evaluated the capacity of the ESP, and (b) processes and criteria for the 
authorizer to also conduct capacity evaluations of the ESP’s governance and leadership. 


Priorities and Application Adaptation 


Standard: The authorizer’s application includes requirements that specialized applicant 
types (i.e., charter network applicants, virtual school applicants, or applicants planning to 
contract with an educational service provider) provide additional relevant information, and to 
the extent applicable, the authorizer adapts its application to address identified needs or 
attract desired program types. 


Key Strengths 


A key strength noted by evaluators was authorizer differentiation in their application process 
based on the types of applications they receive rather than a uniform application system for 
all applicants. For example, many authorizers had separate applications or separate 
application criteria and standards for new and existing operations. Experienced operators 
were often required to provide additional data related to their prior school’s academic track 
record and financial and organizational health, and the organizational structure if the 
applicant sought to replicate the model that existed at their prior school (CMO, EMO, 
affiliation, back office network, etc.). Strong practices specifically included the authorizer 
placing a premium on a thorough evaluation of the applicant’s past performance. 
Additionally, for charter applicants planning to partner with an educational service provider 
(CMO, EMO, or ESP), the application required detailed information about that proposed 
relationship, including the terms of the management agreement, funds expended, and the 
ability of the school’s governing board to dissolve the relationship should performance 
expectations not be met. 


Evaluators also noted that authorizers who had a policy to encourage the replication of 
existing quality schools and had a streamlined review process—heavily focused on past 
performance—was a significant strength. Those systems included the development of 
specific application materials and criteria for expansion/replication and not requiring the 
same depth of descriptions for many application elements required by start-ups (e.g., 
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evidence of past academic and organizational performance replaced application 
descriptions of the school’s financial and organizational systems). 


Key Weaknesses 


Evaluators noted several cases of authorizers failing to request or thoroughly evaluating 
academic performance data from existing school operators. Some authorizers did not 
perform the necessary due diligence on management organizations, academic performance 
evaluations specifically. 


Another key weakness noted by evaluators was the use of the same application and 
evaluation criteria for all applicants, regardless of application type. Examples include 
authorizers that had not developed specific application questions and evaluation criteria for 
virtual school or alternative education campus applicants, and authorizers who did not 
differentiate their application systems for start-ups from replication applications. 


Another area of weaknesses included authorizers failing to communicate application 
priorities clearly to applicants. This weakness was distinct from authorizers that intentionally 
did not adopt application priorities. In cases where authorizers did have priorities, evaluators 
noted that those priorities—for example, seeking schools for specific neighborhoods or grade 
configurations—were not clear in their application materials, RFPs, or other means of making 
those priorities public. 


Decision Alignment 


Standard: The authorizer makes application decisions that are informed by and align with 
documented evidence and analysis of the extent to which the plan satisfies approval criteria 
and the extent to which applicants demonstrate strong preparation and capacity to establish 
and operate a quality charter school. 


Key Strengths 


The clearest strength that surfaced from the evaluations was the processes and materials 
that authorizing staff provided to board members. Many authorizing staffs provided board 
members with detailed, written documents that laid out the strengths and weaknesses of 
applications, justification for their decisions, and evidence supporting those decisions. 
Those documents well prepared board members with the key information needed to make 
decisions. In addition to these reports being timely and detailed, evaluators noted strengths 
relative to boards having the ability to ask follow-up questions of staff, both leading up to 
and at public decision-making meetings. The result of these strengths in practice was a high 
degree of (although not always complete) alignment between staff recommendations and 
board decisions. 


Another key strength was specific to authorizing staff’s use of appropriate data and 
evidence. Evaluators noted that detailed justifications included with evaluation rubrics were 
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often used by board members and applicants themselves to ensure that decisions were 
merit-based. 


Key Weaknesses 


Though evaluators did note that board decision should often align with staff 
recommendations, they also noted instances where board decisions did not align for 
reasons other than the merit of the application; this is distinct from cases where staff 
recommendations and board decisions did not align for differential weighting of key merit 
based factors. For example, evaluators cited cases where board member decisions were not 
only misaligned with the staff recommendation but were based on considerations such as 
the applicant opening or not opening a school in a board member’s district, prior business 
dealings between an authorizing board member and a proposed school board member that 
had nothing to do with his/her capacity and qualifications to oversee a charter school, and 
being generally “anti-charter school,” among other examples. 


A lack of strong evidence-based justifications for staff recommendations could contribute to 
non-merit-based decisions. For example, evaluators noted that for one authorizer, 
unstructured approaches that provided little evidence contributed to board decisions not 
based on the merits of the application. In another example, incomplete recommendations 
made by the authorizing staff to the board—that is, not providing a strong justification and 
set of strengths and weaknesses—resulted in board members “reading into” the application 
and not making decisions based on merit. 


Transparency 


Standard: The authorizer has transparent processes for both application evaluation and 
application decision making. 


Key Strengths 


The most consistently cited strength was making all application materials and processes 
publicly available, typically in an easy-to-find location on the authorizer’s website. Parts of 
the application system noted as strengths in transparency included posting the application 
requirements and processes, a detailed timeline for the application process, and application 
checklists for the applicant to ensure all necessary materials are submitted. In addition, 
evaluators noted strengths related to making public the method of evaluation, the 
evaluation rubric, and samples (or complete records) of approved and denied petitions for 
potential applicants to review. 


The use of annual reports to communicate a range of application decisions and school 
performance was also noted as a strength. In addition to a range of school performance 
data (student demographics, attendance, mobility, test scores, graduation rates, 
descriptions of the school’s education model, financial health, and strengths and 
weaknesses), authorizers used the annual report to communicate the number of 
applications received, disposition of those applications, and other application-related 
information (e.g., proposed opening date, location). 
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Key Weaknesses 


The clearest and most consistent weaknesses that emerged from the data were challenges 
related to school performance reporting. Evaluators noted several cases in which 
authorizers either did not report on school performance publicly or did so in an incomplete 
way. They noted that some authorizers did not do the following: 


e State that they will publicly report the performance of schools 
e Publish an annual report of performance of schools 
e Report on schools' performance relative to established criteria 


e Provide schools with formal feedback regarding their performance until renewal 


Examples of this include one authorizer failing to produce two types of recommended 
annual public reports. The first was an annual report on the performance of each individual 
charter school, summarizing its performance and compliance in accordance with the 
expectations of its charter contract. The second was a comprehensive annual report on the 
performance of its charter portfolio as a whole. 


Another key weakness cited was a lack of transparency in the decision-making process 
itself. This included authorizers not being transparent in terms of how they arrived at 
application recommendations and/or not communicating clear expectations to schools. 
Again, transparency is diminished when authorizers do not provide detailed justifications, in 
addition to ratings, for application approval/denial decisions. 
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PERFORMANCE-BASED ACCOUNTABILITY 


The Performance-Based Accountability domain is organized to assess the degree to which 
authorizers have rigorous, appropriate standards by which schools are held accountable for 
results and whether authorizer decisions are made with the intent to maintain high 
standards and protect the students’ and the public’s interests. Four sub-domains comprise 
this domain: 


e Educational Performance 

e Financial Performance 

e Organizational Performance 
e Decision Alignment 


A number of strengths and weaknesses were found in each sub-domain, and those are 
described following a description of overall key findings on performance-based 
accountability. 


Key Findings 


Making Performance Expectations Clear, Rigorous, Actionable, and Aligned: Among the 
most consistently noted findings across sub-domains was the importance of having multiple, 
specific, and clear measures authorizers used for accountability purposes. Of particular note 
was the view that performance measures were binding in some way, meaning they could be 
legally enforced via an accountability plan or performance framework that was in the 
contract between an authorizer and school (or incorporated by reference) or other means by 
which the authorizer had legal standing to enforce performance expectations. It also 
appeared important that authorizers carefully consider their measures and expectations in 
lieu of state/federal accountability systems, more specifically, ensuring that the authorizer’s 
accountability system did not conflict with state/federal accountability systems, and 
ensuring that if the state/federal accountability system standards were deemed too low that 
the authorizer’s systems ensured a higher level of rigor and expectation. 


Frequent Monitoring and Reporting of Financial Performance: A consistent strength noted 
in the data was the frequent attention authorizers paid to financial monitoring and reporting. 
In addition to being clear, comprehensive, and aligned with state and best practice 
demands, authorizers did much more monitoring and reporting on school financial 
performance compared to academic and organizational performance. While some 
authorizers appeared to struggle with relatively low financial performance expectations and 
internal capacity to tell the difference between concerning financial performance data (e.g., 
data that might trigger a conversation with or notice to the school) compared to serious 
short- or long-term financial crises (e.g., data that required an immediate response from the 
school and/or immediate authorizer action), it appeared that many authorizers were keeping 
a very close eye on the financial performance of the schools they oversaw. 
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Fewer Organizational Performance Expectations: Organizational performance expectations 
were a clearly documented weakness among authorizers. There was a noticeable absence 
of organizational performance expectations among authorizers, especially relative to 
academic and financial performance. In addition, when organizational performance 
expectations were included, they were at times not rigorous, not consistently assessed, and 
unnecessarily inconsistent between schools. 


Strong Documentation Aids in Aligned Decision Making between Staff and Board: Strong 
alignment between staff recommendations on high-stakes decisions (e.g., applications, 
renewal) and the final decision the authorizing board made was a noted strength among 
authorizers. Importantly, while alignment did not need to be 100 percent—authorizing 
boards should not be “rubber stamps” for staff recommendations—and it appeared perfectly 
acceptable for board and staff to disagree on some parts of each application, it did appear 
that strong alignment and near-unanimity on final high-stakes decisions was a key strength. 
It further appeared that authorizing staff who provided strong documentation and detailed 
rationales for recommendations were providing board members with clear direction and 
information to make high-quality decisions. 


The remainder of the Performance-Based Accountability domain describes specific strengths 
and weaknesses emerging from the thematic analysis for each sub-domain. It is of note that 
some practices could be considered both a strength and weakness. In those instances, it is 
likely that some authorizers were quite strong in those practices while others were quite 
weak. 


Educational Performance 


Standard: The authorizer holds schools accountable for academic performance using 
objective and verifiable measures established in the charter contract or performance 
framework that address, at a minimum, student achievement, student growth, and post- 
secondary success as the primary measures of school quality. 


Key Strengths 


The most frequently noted strength was using multiple, specific academic measures to hold 
charter schools accountable. The most frequently noted measure was student academic 
growth followed closely by student achievement. Post-secondary readiness, mission-specific 
academic goals, and comparative measures were also frequently noted. Beyond those five 
measures, authorizers varied in academic measures incorporated into their performance 
measures. Evaluators also frequently noted the incorporation of state accountability 
measures and federal accountability standards into academic performance frameworks. 


Authorizers establishing measurable academic performance goals was one of the strongest 
themes that emerged from the data. Those goals were documented in writing, typically in the 
contract, accountability plan, or performance framework. Evaluators frequently noted as a 
strength the use of a common academic performance framework for all charter schools. 
These performance frameworks identified measures, metrics, and performance levels with 
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clear cutoffs. Data sources for each metric were identified in the performance framework or 
contract. AS a consequence, school leaders understood how their schools would be held 
accountable. 


A second strong theme that emerged from the data was alignment between the charter 
contract, district academic expectations (if applicable), the state accountability system, and 
high-stakes decision making. When the authorizer’s academic performance criteria aligned 
with (and was as rigorous as or more rigorous than) the academic criteria used to hold 
district-run schools accountable, especially at the state level via the state’s accountability 
system, academic standards were less prone to be viewed by stakeholders as arbitrary. 


Placing a premium on the academic performance of schools in decision making was noted 
as a strength. Renewal and revocation decision making focused on academic performance, 
board decisions based on academic performance, and closing academically low-performing 
schools were all noted as strengths. 


Key Weaknesses 


Among the more consistent thematic weaknesses was the lack of alignment between 
various documents and authorities. Evaluators frequently noted a lack of alignment in 
performance expectations across schools, with academic measures and goals varying from 
school to school. In some cases, this was a result of schools developing their own goals with 
no review or approval by the authorizer. Evaluators also noted that authorizers lacked 
alignment across the academic framework and renewal framework. Evaluators noted 
performance goals established in the academic framework that were in conflict with state 
performance expectations, and school leaders reported that they didn’t feel that they were 
being evaluated consistently against the requirements of their charter. 


A strong theme that emerged from the evaluations was the lack of clarity of academic 
performance measures and goals. Evaluators noted some authorizers that lacked an 
academic performance framework as well as authorizers that established measures and 
goals, but those measures and goals were unclear or very difficult to measure. In some 
cases, performance cutoffs were unclear, as was the weighting of each performance goal. 
Often, how authorizers used their performance framework to make accountability decisions 
was noted as unclear. 


Evaluators also frequently highlighted authorizers with weak academic performance 
expectations. Authorizers were noted for taking “the floor and making it the bar for renewal.” 
For some authorizers, relying entirely on state and federal accountability to set academic 
performance expectations resulted in low standards. 


Evaluators frequently noted the absence of specific academic measures and goals as a 
weakness. In contrast to the specific goals cited as strengths, the most frequently 
mentioned missing goal was related to post-secondary readiness measures. Evaluators also 
frequently noted the absence of achievement, student academic growth, and mission- 
specific goals. 
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A strong theme evident in the evaluations was related to the importance (or unimportance) 
of academic performance in high-stakes decision making. Evaluators frequently noted 
examples of academic performance not being considered in intervention policy, revocation, 
and renewal decision making. Often, the issue wasn’t the absence of the measures and 
goals, but the impact of performance toward those goals on authorizer decision making. 


Financial Performance 


Standard: The authorizer holds schools accountable for financial performance using 
appropriate near-term and sustainability measures established in the charter contract or 
performance framework as the primary indicators of a school’s financial viability. 


Key Strengths 


Evaluators frequently noted having a financial performance framework with clear, specific, 
and measurable financial performance expectations as a strength. Expectations were linked 
to the state charter law and aligned with state financial requirements. Performance against 
expectations led to rankings. Performance rankings had clear, often numerical, criteria for 
each ranking. 


Evaluators consistently noted that strong financial accountability systems included a well- 
developed financial monitoring system. Evaluators noted more monitoring and reporting 
strengths in the financial performance sub-domain than in any other sub-domain of 
performance-based accountability, i.e., educational performance and organizational 
performance. Authorizers provided schools with financial reporting guidelines, and contract 
requirements were monitored on an ongoing basis. Regular financial reporting was required 
of schools. Authorizers varied in the frequency with which they required school finance 
reporting, though quarterly reporting was reported most consistently. 


A strong theme among the financial performance strengths was the use of both near-term 
and sustainability measures to hold schools accountable for financial performance. 
Requiring an annual school audit was the most frequently noted strength, with audits 
reviewed against authorizer standards. Evaluators also noted the annual submission and 
review of a school’s budget as strengths. A school’s budget compared to actual expenditures 
and revenues was examined, and schools were required to explain items exceeding the 
budget, as well as any budget deficits. Strong practices included regularly examining fiscal 
ratios, cash on hand, and student enrollment/retention. 


Authorizer use of financial performance information to inform high-stakes decision making 
was frequently highlighted in the evaluations. Authorizers had intervention plans in place 
and intervened when necessary. Summative financial evaluation occurred at renewal. 
Financial performance factored into renewal and revocation decisions. Evaluators noted 
when insufficient financial performance was the basis for closures. 
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Key Weaknesses 


One weakness consistently noted by evaluators was weak financial performance 
expectations. Authorizers monitored specific financial indicators but lacked performance 
standards linked to the indicators. Evaluators also noted financial expectations that were 
not specific. For example, an authorizer might have required a school to be “economically 
sustainable,” but failed to define what qualified as economically sustainable. Evaluators 
also noted authorizers with specific, low standards for financial performance, focused on 
having a positive balance in accounts or decreasing unpaid debts. 


Evaluations described authorizers that did not consistently collect financial performance 
information or collected financial performance information only at renewal. Evaluators noted 
school leaders that were unaware of financial reporting requirements and consequently did 
not report financial performance information to their authorizers. 


Many of the financial performance weaknesses identified by evaluators were related to 
specific financial expectations. Evaluators noted authorizers did not hold schools 
accountable for such things as the school’s current debt-asset ratio (assets divided by 
liabilities), total margin (net income divided by total revenue), cash on hand, and enrollment 
variance. 


Evaluators frequently noted that it was unclear how financial performance affected 
authorizer decision making. Authorizers lacked financial intervention policies or lacked 
specific consequences for specific financial deficiencies. The absence of established 
policies was reflected in schools’ lack of awareness of possible interventions and 
authorizers responding only when problems became dire. 


Evaluators consistently noted staffing capacity issues affecting the quality of financial 
oversight. In some cases, authorizers did not have enough staff, or enough staff trained in 
finances, to provide effective oversight. In other cases, financial monitoring responsibility 
was not delegated formally to staff, resulting in a diffusion of responsibility with financial 
monitoring spread across too many people. 


Organizational Performance 


Standard: The authorizer holds schools accountable for compliance with organizational 
performance requirements established in the charter contract or the performance 
framework, including educational program requirements, governance and reporting, 
financial management and oversight, and operational requirements related to students, 
employees, and the school environment. 


Key Strengths 


Having an organizational performance framework that outlined clear organizational 
performance expectations and requirements was noted as a strength. In addition to 
including expectations around governance, other elements noted as strengths included 
facilities requirements, requirements to adhere to applicable law, requirements for special 
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populations (i.e., reporting requirements and adherence to applicable law), enrollment 
process compliance and results, reporting and compliance requirements, student health 
and safety expectations, requirements for the school environment, teacher and staff 
licensing requirements, and financial management reporting requirements, among others. 


Many of the organizational performance strengths identified by evaluators were related to 
the methods used by authorizers to monitor performance. Authorizers varied in those 
methods, including meetings with schools, written inquiries, stakeholder surveys and focus 
groups, attending governing board meetings, online monitoring systems in which schools 
uploaded data, site visits, and an annual report from schools that included reporting on 
organizational performance indicators. 


Strong practice in this area involved more than data collection. Authorizers must use 
organizational performance data to hold schools accountable. The release by authorizers of 
organizational performance ratings annually and reporting back to schools and the public 
was noted as a strength. As was found in the educational performance and financial 
performance areas, strong authorizers had tiered intervention systems that responded to 
school deficiencies, with consequences ranging from notice, to increased monitoring, to 
closure for organizational performance issues. The use of a Summative evaluation of a 
school’s organizational performance in high-stakes decision making by authorizers was 
noted as a strength. 


Key Weaknesses 


A strong theme running through many weaknesses was the lack of attention to school 
organizational performance by authorizers, especially in comparison to educational and 
financial performance. Evaluators noted authorizers without organizational performance 
frameworks, authorizers that did not establish any organizational performance expectations, 
and authorizers that relied on the state educational agency to require, monitor, collect, and 
evaluate organizational performance data. Documentation of organizational performance 
was unclear on the degree to which organizational performance metrics were evaluated 
and/or used to hold schools accountable. It was also noted that authorizers held schools 
accountable for organizational performance inconsistently or for performance without regard 
to school quality. 


Even when authorizers established expectations, they were not rigorous and not consistently 
assessed. One authorizer allowed schools to define their own organizational performance 
expectations, without authorizer approval. Evaluators also noted authorizers with different 
organizational performance expectations defined in multiple documents and not aligned 
with each other. When those were in conflict, it was not clear which document/set of 
expectations should be used to assess organizational performance. 


Many of the weaknesses identified by evaluators in this area were related to specific 
requirements and expectations missing from organizational performance frameworks. 
Those missing requirements and expectations varied across authorizers with no clear 
pattern. Evaluators noted missing reporting or performance requirements for facilities, 
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student discipline, school calendar, enrollment, admissions and lottery, third-party 
management organizations, governance, special populations, staff credentialing and 
background checks, employee rights, educational program fidelity, and conflicts of interest. 


Evaluators noted many weaknesses related to the monitoring of organizational performance. 
Some authorizers lacked written plans for monitoring organizational performance, and 
others lacked school reporting requirements or relied primarily on an annual site visit to 
monitor organizational performance. 


Decision Alignment 


Standard: The authorizer makes accountability decisions that are informed by and align with 
documented evidence and analysis of the extent to which the school satisfies performance 
expectations. The analysis presented to decision makers is of high quality and the merits of 
the decisions themselves show decision making is based on thoughtful analysis ensuring 
that only the charter schools that meet or exceed expectations are in operation. 


Key Strengths 


The most frequently noted strength was alignment between staff recommendations and 
board votes. However, there was strong evidence across evaluations that this alignment was 
not a rubber stamp for staff recommendations and not 100 percent agreement on every 
area of every application. Strong boards engaged in reviewing recommendations and 
supporting documents. Authorizing board members discussed ratings and recommendations 
with each other and the authorizing staff, sometimes disagreeing with staff ratings. Finally, 
evaluators noted as strengths the positive outcomes of decision alignment coupled with 

high expectations: low-performing schools were closed and only high-performing schools 
were renewed. 


Alignment of renewal documents aided in the process of high-stakes decision making. 
Evaluators frequently noted that alignment of renewal documents, renewal criteria, renewal 
rubrics, renewal application ratings, performance frameworks, charter contract clarity on 
renewals, and recommendations aligned with criteria all aided in strong decision making 
among staff and board members. 


An important strength was a comprehensive, detailed, renewal recommendation report 
provided to authorizing board members. Strong reports included a scored application rubric 
with subsection scores; identification of strengths and weaknesses of the applications; and 
a clear, well-supported, and unambiguous recommendation (up or down) to the board. 
Evaluators noted as strengths recommendations that included detailed rationales for 
recommendations based on both quantitative and qualitative information. Evaluators 
described board members reporting that they trusted staff judgment and felt well-prepared 
to vote on the recommendation. 


Evaluators also frequently noted an alignment of purpose at all levels of authorizing staff, 
leadership, and the board as a strength. Evaluators consistently reported decision making at 
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all levels based on the quality of the school, not outside factors. Reviewers, staff members, 
leaders, and board members were focused on the merit of the application and the 
recommendation of the staff. Board members did not vote based on political affiliation or 
hopes that a school would improve. 


Key Weaknesses 


The primary weakness was an absence of focus on school quality. Evaluators described 
formal decision-making documents that did not state school quality as a priority, state policy 
contexts that did not support quality, and board members reporting that their decisions were 
made based on considerations other than quality. For example, evaluators described 
authorizing boards making decisions based on expected revenue or “pity” for the school. 


Evaluators noted significant weaknesses in the renewal recommendation reports provided 
to board members. In some cases, external consultants, not authorizing staff, produced 
reports with little input from authorizing staff. Evaluators also identified multiple examples of 
staff providing minimal information in reports. For example, evaluators noted reports that 
did not include analysis of the strengths and weaknesses of the renewal application, 
progress toward contract expectations, financial performance, or organizational 
performance. Evaluators described authorizing staff that did not communicate school 
performance to authorizing boards, and renewal recommendation reports that provided very 
little information about the academic performance of schools considered for renewal. 
Evaluators also noted as a weakness staff failure to include an explicit recommendation in 
their report to the board. 


Evaluators consistently identified the absence of formal, documented procedures for board 
decision making as a weakness. This weakness was related to board members feeling 
insufficiently trained on charter school authorizing. Evaluators described boards that 
provided little justification for their decisions or completed a decision-making rubric that 
reflected a prior deliberated decision. Evaluators also reported a lack of clarity regarding the 
decisions board members should weigh in on (e.g., whether or not board members should 
be involved in intervention decision making). Evaluators also noted board decision making in 
conflict with the documented decision-making criteria. 
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PERFORMANCE MANAGEMENT SYSTEMS 


The Performance Management Systems domain is organized to assess the degree to which 
authorizers have effective systems for establishing and monitoring school performance 
expectations and for holding schools accountable as necessary to protect student and 
public interests. Seven sub-domains comprise this domain: 


° Contracting 
° School Openings 
° Ongoing Monitoring 


° School Intervention/Revocation 
° Renewal 

° Closure 

e Transparency 


A number of strengths and weaknesses were found in each sub-domain, and those are 
described following a description of overall key findings on performance management 
systems. Immediately after the sub-domain on Ongoing Monitoring, a description of 
strengths and weaknesses for site visits is also provided. 


Key Findings 


Alignment Across Monitoring and Accountability Systems: A significant theme evident 
within and across each sub-domain was alignment of the systems used to hold schools 
accountable. Evaluators noted misalignment between expectations outlined in the charter 
contract and renewal expectations, as well as expectations outlined in the charter contract 
that were not required in practice. Strong systems clearly aligned contract expectations with 
ongoing monitoring. Schools knew exactly what the authorizer was monitoring and why. 
Similarly, because authorizer intervention policy was aligned with contract expectations, 
schools knew what things would and would not lead to authorizer intervention. Renewal 
criteria were linked to the charter contract, any previous criteria established by the 
authorizer, and the state’s charter school law. As a school moved through its charter term, 
authorizers monitored progress toward performance expectations and frequently reported 
on progress to the school and public. When systems were aligned, there were “no 
surprises.” 


Staffing Capacity and Consistency: Having dedicated staff that managed specific parts of 
the authorizer’s performance management system appeared to be particularly important. 
Internal staff either managed the systems directly, or in some cases, managed external 
consultants who provided targeted support for the authorizer’s system. Importantly, a 
notable strength was alignment between staff operating the authorizer’s monitoring and 
intervention system with a preference for the same person or people managing both. Such a 
system provided schools with a monitoring-intervention process that seemed to increase the 


@ N a Cc S a Qualitative Analysis of Authorizer Evaluations 29 


chances that corrective actions were aligned with findings from ongoing monitoring. 
Importantly, staff consistency related to performance management appeared to be 
particularly important. Frequent turnover was noted as resulting in a host of negative 
experiences for schools and authorizers, including interventions that inappropriately varied 
from school to school based on the approach of whoever was running the system at that 
time. 


Maximizing Autonomy and Providing Clarity in Changes to School Systems: A very clear 
process that made explicit what a school must do to make changes to its operations was 
noted as particularly critical. Authorizer practice that clearly differentiated between school 
operations that required formal approval by the authorizer (typically through the amendment 
process) and those that required authorizer notification (or in some cases, no notification) 
was noted as particularly strong. Very importantly, evaluators noted that strong practices in 
this area included authorizers not only making those expectations clear but also limiting 
areas that required formal approval in order to maximize school autonomy, particularly in 
their educational program, financial decision making, and human capital decisions. 


The Impact of Formal, Written Policies and Clearly Defined Systems: The presence or 
absence of formal policies and systems to hold schools accountable was cited as a critical 
strength and weakness across the sub-domains of the Performance Management Systems 
section. The absence of formal, written policies led to confusion regarding what was, and 
what was not, a material performance expectation the school would be held accountable for. 
In pre-opening systems, some authorizers had allowed schools to open when they were 
clearly not ready—and some of those schools subsequently failed and closed in the first 
year—but the lack of authorizer systems to monitor and set non-negotiable requirements 
was not in place to prevent such failure. Without formal intervention policies, schools and 
authorizers were confused about when and how authorizers would intervene. Without formal 
renewal and closure policies, authorizers might neglect to monitor performance over the 
course of the charter term or have weak non-renewal cases/evidence supporting those 
decisions, making an appeal of a non-renewal decision more likely to be successful. 


The remainder of the Performance Management Systems domain describes specific 
strengths and weaknesses emerging from the thematic analysis for each sub-domain. It is of 
note that some practices could be considered both a strength and weakness. In those 
instances, it is likely that some authorizers were quite strong in those practices while others 
were quite weak. 


Contracting 


Standard: The authorizer executes a charter contract for each school that clearly articulates 
the rights and responsibilities of each party. 


Key Strengths 
Evaluators provided a number of consistently cited strengths, including the following: 
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Contracts and contract negotiation. The practice of signing a contract with each charter 
school overseen was a basic yet important strength. Authorizers had a standard contract 
that was customized for each school, with areas open for negotiation clearly documented by 
the authorizer. Authorizers with strong practices had established frameworks and timelines 
for contract negotiation. After approval of the application, authorizers met with schools to 
negotiate the contract. Contract execution was timely. School leaders reported being 
positive about the contract negotiation process. The school and authorizer maintained 
written and fully executed contracts. 


Autonomy. A strong theme running through the evaluations involved the relationship 
between the material terms of the contract and school autonomy. Evaluators consistently 
noted that strong contracts were limited in scope as to (a) what was required by state law 
and (b) what was necessary to hold a school accountable. Contracts should be specific on 
those items but not so specific that school autonomy is inappropriately limited. Strong 
contracts appeared to be independent from the initial charter school application but might 
incorporate relevant parts of that application. Strong contracts also appeared to be very 
explicit about what constituted a material change in the contract—and thus required 
authorizer approval prior to making that change—but limited those elements so that schools 
had maximum autonomy in their educational program, financial decision making, and 
human capital systems at a minimum. This ensured that the process for requesting changes 
to material items were not administratively burdensome and were addressed very quickly. 


Clear, measurable, and specific performance expectations. The most frequently cited 
contracting strength identified by evaluators was the inclusion of performance expectations 
in each school contract. Academic, financial, operational, and compliance expectations were 
outlined and the terms of measurement were identified. While the authorizer maintained 
strong uniform performance expectations, there was also an explicit opportunity (and in 
some cases, requirement) to tailor performance expectations to the school’s unique mission 
and vision. 


Specific contract elements. Evaluators noted that strong contracts included the following 
elements in addition to performance expectations: the rights and responsibilities of the 
school and the authorizer; the right of the school or authorizer to mutually terminate or non- 
renew; school mission statement; an educational program description; charter term length; 
the requirement that the school be a non-profit organization; the legal status of both parties; 
terms for notice, waiver, severability, assignment, merger, indemnification, survival, 
termination, and dispute resolution; the physical location of the school; financial 
requirements; governance body requirements, including number of board members, board 
roles, and responsibilities; and requirements for management company contract approval. 
This is not to say that other contract elements are not necessary, but reviewers noted these 
elements consistently as strengths. 


Contract amendment. Strong contracting policies included clear, documented processes for 
contract amendment. The contract described changes that require notification versus 
amendment. Substantive/material changes required authorizer approval, while non- 
substantive amendments took effect automatically. All amendments required mutual 
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agreement by the authorizer and the school. Evaluators cited as strengths many specific 
contract elements that should require amendment, including changes in school name, start 
date, facility location, enrollment, grades served, mission, educational philosophy, articles of 
incorporation, and governance structure. Evaluators also cited as strengths specific contract 
elements that should require only notification, including certain changes regarding 
personnel, governing board members, management companies, and other programmatic 
changes. 


Key Weaknesses 


The strongest weakness that emerged from the data was a lack of formal contracting 
policies and procedures. Evaluators noted authorizers who did not sign formal, legally 
binding contracts with the schools. Evaluators also noted that some authorizers lacked a 
standard contract for all schools and in some cases used only the charter application as the 
contract. Some authorizers also lacked formal contract compliance review and document 
processes for compliance monitoring. The absence of formal contracting policies and 
procedures was often associated with contract delays and potential legal challenges to 
school closure. 


Evaluators identified a variety of weaknesses related to performance expectations: 


e Performance expectations were not differentiated from the state’s accountability 
system. Instead, the authorizers relied solely on the state’s accountability system for all 
public schools, which often resulted in an inappropriate performance “floor” and lacked 
expectations on finances and operations. 


e Performance expectations that contradicted each other within the contracting 
documents. Sometimes, performance expectations differed within an authorizer’s 
documents (e.g., the accountability plan’s performance expectations differed from the 
state’s accountability system even when the authorizer’s expectations included both 
systems for school accountability). 


e Performance expectations were not specific enough to be enforceable. For example, 
evaluators noted multiple cases where academic, student demographic, or operational 
expectations were undefined or insufficiently defined. 


e Performance expectations were so low that they were easily achieved or so high that 
they were impossible for the school to achieve. 


e Requirements were outlined in the charter contract that were not required in practice. 


Among the many authorizers that did sign contracts with the schools they oversaw, some 
failed to include specific, important provisions in their contracts. Missing contract provisions 
included performance expectations for short-term versus full-term renewal; a description of 
the charter amendment process, including what constitutes a material change in the 
contract; processes and criteria for charter replication, expansion, and multi-site charters; 
admissions and enrollment policies; an outline of authorizer versus school responsibilities; 
and assurances regarding the legal status of the signatories. 
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Evaluators frequently noted as weaknesses contract elements related to the charter 
amendment process. Some authorizers did not describe the amendment process in the 
contract, used an ad hoc process, or were unclear about what constituted a material change 
to the contract. This could result in immaterial program changes that required amendments, 
as well as the opposite—what should have been a material change to a charter contract 
required only notification. 


School Openings 


Standard: The authorizer ensures that approved schools are prepared adequately for 
opening. 


Key Strengths 


Pre-opening checklists. The strongest theme that emerged from the thematic analysis of 
school opening strengths was authorizers having a pre-opening handbook or checklist that 
described pre-opening tasks. Strong handbooks included specific requirements to be met by 
the school, along with clear deadlines for task completion. Individuals responsible for 
completing requirements were specifically identified and authorizers provided sufficient time 
between charter approval and school opening to complete all requirements. 


Pre-opening requirements. Evaluators noted a variety of specific pre-opening requirements. 
The two most frequently reported requirements were related to enrollment and facilities. 
Authorizers set specific enrollment targets and timelines by which those targets must be 
achieved. Enrollment was regularly monitored and reaching enrollment targets was non- 
negotiable. Similar to enrollment, authorizers set target dates for obtaining facilities. In order 
to open, new charter schools were required to obtain both proof of lease or ownership and a 
certificate of occupancy for the facilities. Evaluators also identified as strengths specific 
requirements related to hiring, governance, finances, and community engagement. 


Monitoring pre-opening requirements. Having systems in place for assessing pre-opening 
requirement completion was noted as an important strength. Authorizers used a variety of 
methods to assess requirement completion, ranging from regular face-to-face meetings and 
phone calls to formal progress assessments and regular site visits. Site visits appeared to 
be an essential component of the pre-opening process. 


Authorizer support. Among the very few times evaluators noted that authorizers providing 
intensive support was a strength was during the pre-opening process. Such support went 
beyond providing a checklist and holding schools to that checklist. In addition to a formal 
process for evaluating progress as previously described, authorizer practices also included 
such things as collaborating with charter school support organizations to support school 
opening, providing explicit informational and step-by-step resources for schools, and in some 
cases advocating on behalf of schools should pre-opening challenges arise (e.g., facility 
zoning issues, obtaining a certificate of occupancy, cutting through internal district 
“bureaucracy”). 
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Holding schools accountable. Strong authorizers held schools accountable for meeting pre- 
opening requirements. School opening was conditional on meeting requirements. Some 
authorizers did not sign a contract with the school until pre-opening requirements were met. 
Authorizers delayed openings or, in Some cases, revoked charters for failure to meet pre- 
opening requirements. 


Key Weaknesses 


Many of the challenges associated with school opening processes are mirror images of 
strengths noted previously: 


Pre-opening checklists. Evaluators described authorizers who lacked any documented pre- 
opening policies or processes. Some authorizers appeared to provide very little oversight 
between charter approval and school opening. Evaluators noted authorizers without pre- 
opening checklists and authorizers with checklists but no specific requirements. Evaluators 
also highlighted authorizers whose pre-opening requirements included tasks not required by 
law. Evaluators described authorizers that lacked pre-opening timelines and deadlines. In 
some cases, authorizers with pre-opening timelines did not provide sufficient time between 
charter approval and school opening. 


Pre-opening requirements. A significant weakness evident in the evaluations related to the 
specificity and rigor of pre-opening requirements. Specific areas of weakness included 
authorizers lacking enrollment checks, pre-opening governance requirements, hiring checks, 
and purchasing checks, and a requirement for schools to obtain certificates of occupancy. 


Monitoring pre-opening requirements. Evaluators identified authorizers who did not have 
systems for assessing pre-opening requirement completion. Authorizers did not regularly 
monitor progress toward requirement completion and did not communicate areas that 
required resolution before school opening. Some authorizers failed to monitor governing 
board activity before opening. Some authorizers had onsite reviews of requirement 
completion, but the review was insufficient or informal. 


Holding schools accountable. Authorizers with strong pre-opening requirements could still 
fail to hold schools accountable to those requirements. Evaluators noted instances where 
pre-opening requirements were not linked to the charter contract, where the parties 
responsible for completing requirements were not identified, and where schools were not 
held accountable to target deadlines. Some authorizers allowed schools to open without 
completing requirements, including failure to reach minimum enrollment targets. Evaluators 
described authorizers with schools whose openings were delayed or never occurred because 
of failures to meet pre-opening requirements. In some instances, delays were noted as a 
weakness because authorizers “pulled the plug” on school opening a day or two prior to 
opening when material weaknesses could have been identified weeks or months in advance 
with better pre-opening systems. 
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Ongoing Monitoring 


Standard: The authorizer has an effective process for monitoring education, financial, and 
organizational performance of the schools it authorizes. 


Key Strengths 


Alignment between authorizing monitoring systems and expectations of schools was the 
strongest theme that emerged from the data. Strong systems had great clarity on what was 
being assessed, how it was being assessed, and the consequences or recommendations 
that would ensue from ongoing monitoring systems. Moreover, how the system works was 
documented in writing, typically in the charter agreement, performance framework, or other 
document easily understood and accessed by both the authorizer and school. Importantly, 
the written system clearly specified where the authorizer’s authority to monitor school 
performance comes from (e.g., state statute, charter contract, performance framework). In 
these systems, schools knew exactly what the authorizer was looking for—and importantly 
what they were not looking for—and there were “no surprises.” 


A second theme was that a strong feedback loop existed between the authorizer’s 
monitoring system and schools. Either immediately or shortly after monitoring activities 
occurred, schools received feedback documenting areas of strength or concern. Whether it 
was through site visit reports, informal conversations with schools, compliance reports, or 
other feedback systems, a noted strength was that schools frequently had great clarity on 
where they stood relative to authorizer expectations 


There appeared to be a wide variety of methods and tools authorizes used to monitor school 
performance. Site visits were by far the most frequently mentioned ongoing monitoring 
method noted by evaluators (See the next section for additional data on strengths and 
weaknesses of site visits). Electronic software data collection, annual reports, attendance at 
governing board meetings, an annual calendar/schedule of reporting requirements, and an 
observation checklist were other frequently mentioned monitoring tools used by authorizers. 


Evaluators noted that in addition to educational, financial, and organizational performance 
monitoring, authorizers also monitored a number of other compliance areas. It appeared 
that much of this additional monitoring was driven by state law or policy, and included 
authorizers monitoring such things as health and safety (especially facility and school safety 
plans), school lottery processes, proof of adequate insurance, qualifications of teachers, 
special education and English language learner compliance, assessment practices (i.e., 
state test confidentiality procedures), and conflict of interest documentation. 


Having dedicated staff responsible for running the oversight system or overseeing external 
consultants who provide data for all/parts of the system was also noted as an important 
strength. The strongest models utilized this person (or team of people) to manage both the 
ongoing monitoring and intervention systems. That combination provided schools with a 
monitoring-intervention process and appeared to increase the chances that school 
corrective actions were aligned with findings from ongoing monitoring. 
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Key Weaknesses 


The strongest weakness that emerged from the data was a lack of a monitoring policy or 
clearly specified expectations, and misalignment between monitoring systems and 
expectations. In these cases, there were no framework, protocols, or other criteria that 
guided the work of the authorizer and provided clarity to schools. Authorizer activities were 
reactionary to any “problem” that arose, as defined by the authorizer. In addition, schools 
had no idea what they were being held accountable for and were cited for problems they did 
not know they could be held accountable for. 


A second frequently mentioned and strong theme was the implication of having limited staff 
capacity and training to effectively manage ongoing oversight systems. In some cases, 
authorizers did not have many (or in a few cases, any) people dedicated to ongoing 
oversight. In addition, authorizers with significant turnover in monitoring system 
management resulted in tremendous inconsistency. As noted by evaluators, frequent 
turnover resulted in schools not knowing who to submit information to. It also resulted in an 
unstable and always-changing system. Stated differently, implementation was idiosyncratic 
based on the perceptions and skills of whoever was running the system at the time. 


A Special Note on Site Visits 


There is disagreement in the field about the utility of site visits. Some authorizes maintain 
site visits provide little additional information that cannot be obtained through other 
methods, provide for an unnecessary additional burden on schools and stakeholders, and 
run the risk of infringing on school autonomy. Others maintain that valuable information— 
that cannot be obtained any other way—is gathered through site visits and that engaging 
stakeholders face-to-face is a critical authorizing function. 


While it is beyond the scope of this report to comment on the utility of site visits, evaluators 
noted both strengths and weaknesses in site visit processes. Four key themes for each are 
described in Table 1. 


Table 1: Summary of Site Visit Strengths and Weaknesses 


Key Strengths Key Weaknesses 

Interviews with many stakeholders, Examining inputs, including teacher 
including parents, board members, staff, questioning techniques, classroom 

and students management, lesson materials 
Observations of board meetings, Number and intensive structure of site 
classrooms (for consistency in charter visits are not sustainable (e.g., as the 
agreement only), and facilities, especially number of schools in an authorizer’s 

the cafeteria and restrooms for portfolio grows, current system is too labor- 
health/safety checks intensive) 
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Post-visit information sharing from site visit | Authorizing staff without expertise or 
protocol is aligned with written expectations | training in site visits almost always results 
in examination of educational inputs 
Detailed report—including a summary Informal feedback and no clear assessment 
assessment—is returned to the school in a of how the school is doing 

timely manner 


School Intervention/Revocation 


Standard: The authorizer has effective policies and practices for school intervention and 
revocation and conducts merit-based interventions, including revocation where appropriate, 
in response to clearly identified deficiencies in the school’s record of educational, 
organizational, and/or financial performance. 


Key Strengths 


The greatest strength was a clear, written intervention system aligned with performance 
expectations typically included in the charter contract, performance framework, and/or 
accountability plan. Similar to findings from the ongoing monitoring system, schools knew 
not only what things would—and would not—cause a formal reaction by the authorizer, but 
also what type of intervention (including revocation in extreme cases) would be applied by 
the authorizer. 


It appears that most authorizers used a common rating system for all schools in their 
portfolio for educational and financial performance areas. Schools were typically placed in a 
performance category—most typically one of three to five performance tiers—based on 
annual performance metrics. Each of those tiers had a clearly pre-specified set of required 
activities and performance improvement expectations written into the accountability plan. 


Authorizers also typically had an intervention system for compliance with and fidelity to state 
law and the charter contract, and at times these items were separate from the “tiered” 
intervention process. While there was a fairly common system authorizers used to notify the 
school of non-compliance, what authorizers required of schools to remedy non-compliance 
appeared to be customized and aligned with the identified problem. Typically, authorizers 
sent a letter to the school leader and board chair that included (a) the reason for non- 
compliance; (b) expectations and/or targets to address the findings; (c) the expected 
timeline for a successful remedy; (d) how the authorizer would verify progress; (e) 
consequences for continued non-compliance; and (f) an opportunity for the school to 
respond to findings. In many cases, item (b)—expectations and/or targets demonstrating the 
problem has been remedied—was aligned with the problem both in scope and substance. 
The expectations for remedying non-compliance with facility health and safety issues were 
different than non-compliance in submitting timely financial data to the authorizer. Some 
infractions, such as cash flow problems, enrollment problems, or some student 
health/safety problems, triggered an immediate “intent to revoke” letter. 
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Additional authorizing practices associated with intervention plans varied. They included 
such things as prohibiting schools from replicating or expanding until problems were 
sufficiently addressed; conducting additional site visits, sometimes unscheduled; additional 
and/or more frequent reporting requirements; more frequent meetings with school 
leadership and board members; or, in Serious instances, requiring the school to appear 
before the authorizing board. 


The degree of authorizing engagement with school intervention plans also varied widely. 
Evaluators noted that some authorizers were very “hands on,” doing such things as 
prescribing how problems had to be fixed (e.g., step-by-step instructions for what the school 
must do to fix the problem), and approving—rather than accepting—the school’s intervention 
or corrective action plan. Others were not prescriptive in how schools fixed non-compliance 
issues or were prescriptive only for legal and/or financial challenges. 


Key Weaknesses 


Evaluators noted three key weaknesses in authorizer intervention/revocation systems: 


Lack of an intervention/revocation policy. The strongest theme was a lack of an 
intervention or revocation policy. Evaluators noted that a number of authorizers did not have 
a formal policy that grounded and directed their actions to intervene in schools. In these 
instances, there was great confusion over what steps the authorizer should take to 
intervene and how that process should proceed, resulting in (a) authorizers intervening for 
very “small” things, (6) authorizers not intervening for significant infractions/challenges, or 
(c) authorizer intervention being idiosyncratic, driven by whatever external factors happened 
to come up (newspaper story, parent complaint). 


Inadequate intervention/revocation policy. Some authorizers had intervention/revocation 
policies, but those policies were lacking in key areas. In these instances, authorizer actions 
were very informal and reactive. For example, evaluators noted the following: 


e Some authorizers had no guidance or specificity on what would constitute corrective 
action. 


e Some authorizers had no criteria for intervention that were explicitly based on state 
statute, the charter contract, performance framework, or other written expectations 
for school performance. 


e Some authorizers used criteria for intervention that were so broad that it was very 
unclear what actions or performance could trigger authorizer intervention/revocation. 


Inadequacies in intervention/revocation policy implementation. In other instances, 
evaluators noted challenges with how intervention/revocation policies were implemented by 
staff. For example, evaluators noted the following deficiencies: 


e Not adhering to a specified timeline for remedy 


e Not tracking progress toward remedy 
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e Not enforcing consequences, even if specified in writing 


e Not differentiating between very serious problems that should cause notice of 
revocation and moderate to serious problems that should call for an appropriate 
timeline for remedy 


The most significant consequences of challenges associated with an inadequate or absent 
intervention/revocation policy and implementation were associated with a school’s record 
when the school’s charter contract was set to expire, i.e., charter renewal. Evaluators noted 
that schools may not have known what things would be a part of their record evaluated for 
renewal, and/or authorizers would not be able to produce a strong enough case for non- 
renewal, resulting in non-renewal decisions being overturned by an appellate body in some 
instances. 


Renewal 


Standard: The authorizer runs a well-structured renewal process, including clear 
requirements, a meaningful opportunity for the school to present information and respond 
to the authorizer’s findings, clear communication, and prompt notification of decisions. 


Key Strengths 


Renewal criteria. Charter school renewal criteria were frequently identified by evaluators as 
strengths. Authorizers had clear, documented renewal criteria to guide renewal decision 
making. Renewal criteria were linked to the charter contract, any previous criteria 
established by the authorizer, and the state’s charter school law. Academic and financial 
performance criteria were among the most frequently cited strengths in the renewal section 
of the evaluation. Other criteria noted as strong included organizational, operational, 
governance, and compliance performance criteria. Many authorizers differentially weighted 
criteria with some criteria contributing more to the renewal decision than others; for 
example, student outcome data consistently received higher weight than other areas. 


Renewal applications. Most authorizers required charter schools to submit an application or 
written request in order to be considered for renewal, and the scope and breadth of the 
renewal application varied across authorizers. Strong renewal applications included two 
primary components: (1) An analysis of the school’s academic, operational, and financial 
performance over the charter term; and (2) proposed goals for the new term. Some 
authorizers required charter schools to submit very little information other than what was 
required by state law, as they believed they had already collected a lot of data throughout 
the charter term. Others had comprehensive requirements that covered diverse areas of 
school performance. Some authorizers also required a description of any proposed changes 
to the school program, along with a description of how the school would achieve success in 
the next term. 


Renewal evidence gathering. Authorizers with strong renewal processes had systems in 


place to monitor the ongoing performance of each charter school and communicated 
performance assessments to the schools in their portfolios. Authorizer use of site visits to 
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evaluate school performance and educate schools about the renewal process was 
frequently cited as a renewal strength. The scope of site visits tended to vary, depending on 
the ongoing performance and size of the school. Some authorizers also conducted renewal 
applicant interviews and sought community input regarding the charter renewal. 


Renewal timeliness. Evaluators noted as strengths renewal processes that were conducted 
in a reliable and timely manner. Strong charter school renewal processes began early in the 
final school year of the charter contract. Authorizers made initial renewal decisions and 
informed schools in a timely manner, typically near the mid-point of the school year. Schools 
were given sufficient time to respond to the renewal recommendation or decision, aligned 
with the authorizer or state appeal process (if applicable). The authorizer made final renewal 
decisions well before the expiration of the charter term, especially if the decision was to non- 
renew so that parents and families had sufficient time to access another school option. 


Renewal performance analysis and report. As the end of a charter school’s term 
approached, authorizers typically conducted a renewal performance analysis. For some 
authorizers, this consisted of an evaluation and scoring of the renewal application by 
authorizing staff and/or the authorizing board. Other authorizers conducted a school 
performance framework review. External reviewers—either in other parts of the authorizing 
organization (e.g., other district staff) or external to the authorizer altogether—contributed to 
components of the review. Authorizers produced a cumulative renewal report that 
summarized the performance of the school over the term, provided a written explanation of 
the grounds for renewal or non-renewal, and included a renewal recommendation to the 
authorizer’s board. Renewal reports were shared with schools during the renewal process. 
Strong renewal reports and recommendations were focused on school outcomes rather than 
school efforts/input. 


School response performance analysis. Authorizer policies that provided schools an 
opportunity to respond to or appeal the renewal recommendation or decision were cited as a 
strength. Authorizers varied in how they provided this opportunity. Some authorizers used 
mandatory public hearings. Others scheduled hearings by school request. Some authorizers 
permitted schools to present their case at a community meeting before the board vote. 
Others allowed schools to submit a plan of correction and re-evaluated renewal decisions 
after reviewing the plan of correction. 


Renewal decision making. Charter renewal decisions were linked to the charter contract, 
aligned with a school performance framework, and based on rigorous performance 
outcomes. Renewal decisions included an authorizing board vote, and board decisions 
consistently aligned with authorizing staff recommendations. 


Key Weaknesses 


Low standards or renewal for future promises. Among the more important themes emerging 
from the data were examples of authorizers and/or authorizing boards either (a) not 
following their own renewal standard criteria, or (b) renewing school contracts because of 
promises for future improvement. In some instances, community and/or political pressure 
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caused authorizer recommendations and board approvals not to adhere to clearly defined 
renewal performance expectations. In addition, multiple cases were identified where a 
school with a consistently poor performance record—particularly in academics—was renewed 
because of either recent administrative change the year of or just prior to renewal (e.g., 
changes/declination of relationships with management companies, changes in school 
leadership) or promises for future academic success as a part of the renewal application. 


Documented criteria absent or inadequate. Nearly every authorizer evaluated had 
established renewal criteria, but evaluators noted a diverse range of weaknesses related to 
those criteria. Among the more critical weaknesses noted were renewal criteria that differed 
from the criteria outlined in the charter contract or performance framework, or when school 
performance expectations changed over the course of a charter term without negotiation. A 
second critical weakness was renewal criteria focused too heavily on compliance or 
measures focused on school inputs instead of outcomes. Evaluators also consistently noted 
other weaknesses, including authorizers relying solely on state accountability system ratings 
for renewal decision making; a lack of clarity on criteria that would be used for accountability 
purposes (state criteria, authorizer criteria, or both); and in some cases, the absence of 
clear criteria led to renewal processes being implemented inconsistently across schools, 
with each school being held to different performance standards. 


Renewal timelines. Evaluators frequently noted weaknesses related to renewal timelines. 
Noted challenges included renewal timelines too short for schools to respond to renewal 
findings, and renewal decisions announced too late in the school year, thus limiting student 
and family options if a school was not renewed. 


Evidence-gathering challenges. Evaluators identified a variety of renewal weaknesses in the 
systems authorizers used to monitor school performance: 


e The lack of systems to monitor and document school performance over the charter 
term 


e Waiting until the end of the charter term to gather school performance information, 
oftentimes resulting in inaccurate or disputed information 


e Gathering performance information unrelated to renewal criteria 


Renewal performance analysis and report. Some authorizers lacked established protocols 
used to reach renewal decisions or had no method to translate performance information 
into renewal decisions. In some cases, renewal decision-making protocols allowed 
composite renewal scores to mask issues in particular performance domains. Some 
authorizers didn’t provide cumulative renewal reports to schools, and some that did may not 
have provided either sufficient detail regarding school performance or a renewal 
recommendation in the report. In multiple evaluations, evaluators noted as a weakness 
authorizers that did not provide schools the opportunity to respond to the cumulative record 
presented by the authorizing staff or to appeal an authorizer decision. 
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Closure 


Standard: Following non-renewal, revocation, or voluntary return of the charter, the 
authorizer has an effective plan for and ensures the orderly closure of a school. 


Key Strengths 


Authorizer communication of closure to schools, parents, and other key stakeholders was 
noted as a critical strength. Before the decision, authorizers with strong practices regularly 
communicated expectations to schools. After the decision, authorizers provided timely 
notification of the decision, and had processes in place to communicate the closure decision 
and process to school leaders, staff, families, stakeholders, and the media. 


Evaluators frequently noted strengths related to closure plans. Closure plans should be 
included in the charter contract and provide schools a list of closure tasks, along with a 
timeline for completion. Strong closure plans identify the roles of the authorizer and the 
school. Plans should be specific and describe the orderly wind-up of school operations and 
finances. Ensuring that the disposition of public property and funds and attending to assets 
post-closure decision were also noted as critical for either the authorizer to address directly 
or ensure that other regulatory bodies successfully executed those functions. Strong plans 
provided for the securing of school and student records and transfer of those records to 
other schools for a number of years beyond the school’s closing. Authorizer processes 
working closely with families to ensure new school placement were also noted as important 
strengths. 


Authorizer capacity and staffing were noted as critical in successful school closures, and the 
degree to which authorizers used internal versus external support varied. Some authorizers 
had staff that led the closure process and executed many closure functions with the school. 
Authorizers that lacked the staff and/or internal capacity to manage closure had dedicated 
resources to hire part-time staff or consultants to oversee the closure process. Some 
authorizers employed part-time staff or consultants in the role of “transition liaison” to 
streamline student placement and financial experts to oversee the financial dissolution of 
the school. 


Key Weaknesses 


Evaluators frequently noted authorizers lacked a formal, comprehensive closure plan to 
ensure orderly closure of a school. Authorizers with weak closure practices may have no 
closure requirements beyond those required by state law/policy. Evaluators noted closure 
plans that were not specific enough to provide actionable guidance to schools, closure 
timelines that were incomplete, out of chronological order, or did not provide adequate time 
for stakeholder action. 


Frequently, evaluators noted specific elements missing from school closure plans. Missing 
critical financial safeguards were noted, including processes for property disposition and 
asset liquidation and the requirement for an escrow account to cover closure costs. 
Evaluators also described closure plans that inadequately addressed community and family 
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needs during the closure process. For example, authorizers lacked guidance for 
communication of school closure to communities and families. Others lacked processes for 
the placement of students post-closure. Weak closure plans did not identify school versus 
authorizer responsibility for closure tasks and did not specify who within the authorizing 
office was responsible for closure oversight. 


Transparency 


Standard: The authorizer communicates to schools and the public clearly and consistently 
regarding expectations for and status of school performance, including formal reporting on 
school performance at least annually. 


Key Strengths 


Both having a school performance framework and clearly communicating that framework 
were noted as transparency strengths. Authorizing staff made efforts to educate boards, 
schools, and the public on performance frameworks proactively and continuously. 


Annual authorizer reports to the public and individually to schools were identified as critical 
tools supporting transparency in charter school authorizing. Authorizers reported both on 
individual school performance and on the performance of its entire portfolio. Reports 
described both expected outcomes and progress toward those outcomes. Elements 
described by evaluators as strengths included student academic outcomes, financial 
performance, enrollment, attendance, mobility, and attrition rates. 


While specific measures in annual reports appeared to be important, strong reports aligned 
with accountability and performance frameworks. For example, providing federal, state, and 
school-specific goals and progress was noted as important to the degree to which the school 
was being held accountable for all of those indicators. Including performance information 
not aligned with performance expectations ran the risk of confusing schools and other 
stakeholders as to what they were being held accountable for. 


The distribution of school performance information by authorizers to the authorizing board, 
parents, and the public was cited as a strength. Authorizers varied in distribution method. 
Some authorizers distributed printed annual reports, while others posted annual reports and 
performance framework results on a website or included school-level performance 
information in online dashboards. 


Authorizers have also developed practices to support transparency at the authorizer board 
level. For example, strong authorizers posted board agendas and supporting documents 
online and streamed and archived board meetings. Other authorizers made efforts to 
support transparency via more traditional means by providing public hearings as part of their 
renewal process and announcing renewal decisions in a public venue. 


Key Weaknesses 


A lack of clarity around school performance expectations was a common theme among the 
transparency weaknesses. What produced lack of clarity varied. Authorizers might not have 
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aligned their performance reporting with performance expectations or renewal criteria. 
Reported charter office, district, and state performance standards may have conflicted. 
School evaluation methods may not have been described or, in some cases, the authorizer 
may have lacked detailed school accountability procedures. 


Evaluators frequently identified authorizers that did not produce a regular comprehensive 
report of school and portfolio performance or did not report any school performance 
information outside of renewal or intervention systems. As a consequence, authorizer board 
members and charter school board members reported that they did not know where schools 
stood in relation to authorizer expectations outside of the renewal process. 


Weaknesses were also noted relative to the methods authorizers used to distribute 
performance information to schools and the public, as well as the completeness of reports. 
Some authorizers lacked a consolidated, centralized system to distribute school 
performance reports. Evaluators frequently noted authorizers that had limited information 
about charter schools on the authorizer’s website. Evaluators also noted problems with 
missing information related to compliance, organizational/operational performance, 
financial performance, and mission-specific goal performance. 
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ORGANIZATIONAL CAPACITY 


The Organizational Capacity domain is organized to assess the degree to which the 
organizational structure and systems support quality authorizing practices and forward the 
authorizer’s mission. Six sub-domains comprise this domain: 


e Strategic Plan 

e Organizational Structure 

e Human Capital Processes and Systems 
e Conflicts of Interest 

e Organizational Budget 


e Leadership and Decision-Making Body 


A number of strengths and weaknesses were found in each sub-domain, and those are 
described following a description of overall key findings on organizational capacity.1° 


Key Findings 


Lack of Organizational Development Systems: Among the more significant and clearest 
weaknesses was the recurring lack of organizational capacity systems. That included the 
absence of strategic plans and associated goals, strategies, activities, and monitoring 
systems, and the absence of staff development systems, including hiring, onboarding, 
professional development, retention, promotion, dismissal, and staff performance feedback 
systems. Moreover, the lack of intentional planning was associated with a lack of long-term 
and strategic financial planning, resulting in authorizers reflexively spending resources on 
short-term activities instead of intentionally on strategic aims and plans. 


Diversity of Purposes: There was tremendous diversity in the mission statements of 
authorizers. From chartering schools to meet only specific community needs to authorizing 
innovative educational options to fostering choice and competition, there appeared to be 
vast differences in why authorizers were involved in charter school authorization. While it is 
beyond the scope of this analysis to assess the degree to which those mission statements 
relate to such things as the kind of schools authorized or the appetite for rapid versus slow 
growth, it could be that the widely diverse population of charter school sectors across the 
country could be related to different intents and purposes of authorizers. 


Sufficient Staffing and Longevity: An insufficient number and quality of staff resulted in 
authorizers missing important indicators and events happening in their portfolios. Most 


10 The Organizational Capacity domain and its six sub-domains were added to NACSA’s evaluation process in 
2014. Thus, excerpts and findings from the Organizational Capacity domain are from 48 percent of the 
sample. 


2) N a C S a Qualitative Analysis of Authorizer Evaluations 45 


authorizers mixed full-time, part-time, and external consultants as part of their 
organizational structure, and there was no consensus on the number of staff needed to 
successfully execute authorizing functions. Findings suggested that, rather than a precise 
staffing number, an adequate combination of staff and external consultants, and enough 
full-time staff capacity to effectively oversee external consultants, appeared to be critical. 
Staff longevity also appeared to be a significant asset. While the evaluations did not indicate 
that longevity should be more important than the quality of work executed by staff, they did 
note that having a core staff in place—typically for five to seven years or more—tended to 
produce strong working relationships and provide schools with consistent leadership and 
engagement. 


Leadership Culture of Rigor: Authorizers’ pride in exacting a high level of rigor in authorizing 
systems was noted as a strength by evaluators. Authorizers did not want to be viewed as 
“anti-charter” by having systems that rendered it impossible (or nearly impossible) for quality 
schools to exist, grow, and thrive, but did express a desire to be known for being difficult to 
get a charter from and not wavering to close schools they deemed not meeting their 
expectations, even when under political pressure to make a different decision. 


The remainder of the Organizational Capacity domain describes specific strengths and 
weaknesses emerging from the thematic analysis for each sub-domain. It is of note that 
some practices could be considered both a strength and a weakness. In those instances, it 
is likely that some authorizers were quite strong in those practices while others were quite 
weak. 


Strategic Plan 


Standard: The authorizer plans well for the future in a way that aligns with NACSA’s 
Principles & Standards. The authorizer uses quality authorizing to forward its mission. 


Key Strengths 


The most consistently noted strength was how authorizers used their strategic plan. Some 
specific activities mentioned included the following: 


e The strategic plan had measurable goals to which staff and board members 
continuously referred. 


e The strategic plan was not static but rather one that staff and board members 
modified as conditions changed. 


e The strategic plan was used to self-evaluate progress on a consistent basis. 
Evaluators also noted that the stronger strategic plans had a vision statement, a mission, 


core values and/or guiding principles, measurable goals, and specific detailed activities 
aligned with achieving goals. 
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While evaluators did not examine the quality of the mission statement, most authorizers did 
have a mission statement, and evaluators listed that mission statement in most evaluation 
reports. It is noteworthy that mission statements among authorizers varied quite a bit. Some 
mission statements made note of authorizing schools designed to serve underserved 
populations; others mentioned authorizing distinct, innovative educational programs; some 
spoke to filling gaps in educational opportunity; some noted a focus on providing resources 
and guidance to schools; and others made explicit mention of having a portfolio of schools 
that outperformed other public schools. 


Key Weaknesses 


The most consistent theme—by far—was authorizers not having a strategic plan, as well as 
having no elements of a strategic plan. Evaluators consistently noted the following: 


e Some authorizers had no specific strategic goals. 


e Some authorizers had no strategies, activities, or timelines, nor were any of these 
aligned with a strategic plan. 


e Even among some authorizers that did have a form of a strategic plan, it was 
incomplete, had no method to measure progress or make modifications toward plan 
goals, or had no measurable goals aligned with the authorizer’s mission. 


Organizational Structure 


Standard: The authorizer purposefully and economically staffs its office to effectively carry 
out its authorizing duties. Staff positions are clearly defined both in policy and in practice. 


Key Strengths 


Evaluators noted vast differences in the number of full- and part-time staff authorizers had 
in carrying out their authorizing duties. From one- to two-person authorizing organizations 
(notable in that the vast number of evaluations comprise authorizing organizations of 10 
schools or more), to authorizers that were very well resourced, strengths were identified 
across staffing systems. While it is beyond the scope of this report to identify the most 
effective number of staff necessary to carry out authorizing duties, it is noteworthy that 
evaluators concluded that core authorizing duties could be carried out via a number of 
different organizational structure arrangements. 


Evaluators noted that most authorizers carried out their authorizing duties via a combination 
of full-time staff, part-time staff, and external consultants. Within this staffing arrangement, 
evaluators noted that having a very clear set of responsibilities among staff and consultants 
was important. More specifically, clearly defining the role of external consultants—and 
differentiating that from the work of full-time staff—was consistently noted as critical for 
effective staffing operations. 
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Key Weaknesses 


The only consistently noted weakness was the lack of staff to carry out even basic 
authorizing duties. As previously noted, authorizers with even small full-time staffs—coupled 
with strong and well-managed external consultants—can effectively carry out authorizing 
duties. Yet, evaluators consistently noted instances of authorizers having an inadequate 
combination of staff and external consultants, and inadequate people/systems to effectively 
oversee external consultants. 


In some instances, evaluators noted the consequences of not having enough staff to carry 
out authorizing duties or to oversee external consultants. They noted that authorizers were 
either (a) missing important things happening in schools such as financial insolvency, and 
(b) paying external consultants with little oversight, which resulted in authorizers receiving 
inadequate or inaccurate information used in high-stakes decision making. 


Human Capital Processes and Systems 


Standard: The authorizer has systems necessary for building and maintaining a strong 
workforce and implements them with fidelity. 


Key Strengths 


Three specific themes emerged related to authorizing strengths in human capital processes 
and systems: 


Staff consistency and development. The most consistently mentioned human capital 
processes and systems strength was the benefit of having many staff members in place for 
a relatively long period of time and a focus on professional development opportunities. 
Evaluators noted that staff longevity—which typically was five to seven years or more— 
appeared to produce strong working relationships among staff and provided consistency in 
oversight and relationships with schools. Evaluations did not indicate that longevity should 
be valued at the expense of quality but did consistently note that longevity was an asset. 
One reason for effective staff consistency was a focus on staff professional development. 
Evaluators noted that strong professional development plans were aligned with both 
professional and personal interests. Moreover, professional development systems were 
explicitly designed to build staff internal capacity to improve and advance the organization, 
resulting in staff retention and promotion in a number of instances. 


Alignment with mission. Staff having knowledge of the authorizer’s mission and their 
specific role in achieving it also emerged as an important theme. Even within the context of 
a wide range of missions adopted by authorizers, evaluators noted that a significant 
strength was having staff who knew and could articulate their organization’s mission and— 
importantly—could describe their specific set of activities and responsibilities in achieving it. 


@ N a C S a Qualitative Analysis of Authorizer Evaluations 48 


Working relationships with the executive director. Evaluators noted that executive 
directors11 appear to come from many different prior professions, including higher 
education, state policy, charter school support/advocacy organizations, philanthropic 
organizations, and traditional K-12 education (teacher, principal). They noted that among 
the most important strengths was the strong relationships they developed and maintained 
with board members, other superiors (e.g., superintendent), and staff. Weekly meetings, 
orientation sessions, creating and supporting professional development opportunities, and 
having “tough” conversations—especially with board members—were all noted as specific 
activities executive directors engaged in to cultivate positive working relationships. 


Key Weaknesses 


The most consistent weakness was the lack of any human capital processes and systems. 
More specifically, evaluators consistently noted that authorizers lacked the following: 


e Systems for retention, promotion, or dismissal 

e Formal hiring or onboarding processes 

e Feedback on professional performance 

e Formal professional development systems aligned with organizational priorities 

e Plans for new board member identification and/or development 
In authorizing organizations that are not designed solely for authorizing, evaluators also 
noted authorizing was a low organizational priority. Consequences for such included (a) 
bureaucratic processes that delayed or dramatically extended the time to hire key staff 
positions, including the executive director; (b) agency hiring rules—such as limits on salaries 
and requiring internal candidate seniority—that limited the organization’s ability to attract 
talent; and (c) approval processes for functions such as attending professional development 


activities that were cumbersome and lengthy, resulting in staff not seeking out or attending 
such opportunities, or paying for them on their own. 


Conflicts of Interest 


Standard: The authorizer operates free from conflicts of interest. 


Key Strengths 


Evaluators noted four specific practice strengths of authorizers designed to operate free 
from conflicts of interest: 


Formal conflict of interest policy. Having a formal conflict of interest policy that (a) was 
written and made public; (b) applied to all people involved in any way with the authorizing 
endeavor, including board, staff, all external reviewers/consultants, and other staff internal 


11 The term executive director encompasses other titles of day-to-day authorizing leaders including charter 
school director. 
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to the organization (e.g., other district office staff); (c) was signed by all parties and, (d) was 
reviewed and updated on a semi-regular basis was viewed as critical for success. 


Decision to have no contractual relationships with schools where money or influence was 
exchanged. Even in places where authorizers could, by law, engage in fee-based services to 
schools or play a role in school governance, evaluators noted as strengths that authorizers 
had intentionally not provided fee-based services or engaged in practices such as taking a 
voting seat on a school’s governing board and/or appointing school governing boards. 


Clear, transparent contacts when resources are exchanged. Evaluators noted that some 
authorizers might not have been able to avoid financial relationships with schools, such as a 
district authorizer that owned school facilities and made those spaces available to charter 
schools they oversaw. In those cases, evaluators noted that authorizers had detailed 
contracts that were very public and transparent, even if not required by law. 


Reduce per-student revenue fees. The majority of authorizers were funded via a per-student 
fee from the schools they oversaw (typically three percent). Evaluators noted that 
authorizers had reduced this fee to avoid the appearance of a conflict of interest, 
particularly to avoid the perception that they opened schools for additional resources and 
failed to close schools due to revenue loss (instead of based on merit in both cases). 


Politically appointed and/or elected board members. In many cases, the governing board 
members of authorizing organizations were elected officials or appointees of elected 
officials. Evaluators noted that strong conflict of interest policies—that required a board 
member to recuse her/himself from any discussion or vote for any applicant or school that 
member had a connection with—could help mitigate conflicts of interest among “politically 
connected” applicants or schools. 


Key Weaknesses 


Having a missing, limited, or incomplete conflict of interest policy was the strongest theme 
that emerged from the data. Policies that (a) applied only to some staff and/or consultants, 
(b) were informal and not written, and (c) required only disclosure of potential conflicts (e.g., 
allowing consultants to have school service/improvement roles with schools they are also 
contracted to evaluate, as long as they disclosed such) were noted as examples of 
problematic conflict of interest policies. 


Evaluators also noted the inherent conflicts of interest that arose when authorizers had 
direct control over school governing boards. Examples included authorizers that appointed 
charter school governing boards and had the authority to change board members at any 
time, and instances where charter schools had no autonomy from the authorizer governing 
board (e.g., the school district’s governing board was also the governing board of the charter 
school). 
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Organizational Budget 


Standard: The authorizer’s budget allows for organizational effectiveness and stability. The 
budget is aligned with the strategic goals and supports quality authorizing practice. 


Key Strengths 


Having a budget that provided sufficient resources for such things as adequate staffing, 
implementation of the organization’s strategic plan, application reviews (especially external 
reviews), ongoing monitoring, and a closure process emerged as a strength from the data. 
When budgets were not fully sufficient, evaluators noted how authorizers sought to fulfill 
core authorizing responsibilities creatively. That included such things as contracting out 
certain functions rather than making the long-term commitment of hiring new staff members 
and seeking in-kind contributions for some authorizing responsibilities, such as having other 
authorizers or internal organizational staff help with application reviews. 


Appropriate oversight and effective execution of the budget also emerged as strengths. 
Specifically, evaluators noted that having a board approve the budget annually was a 
strength, as was staff executing the budget within approved parameters and making 
significant budget adjustments with board approval only. 


Key Weaknesses 


Authorizers who lacked an annual budget or long-term financial plan emerged as one of the 
more consistent thematic weaknesses. Evaluators noted this could also be related to a lack 
of an organizational strategic plan. As a consequence of not having a budget or financial 
plan, authorizers spent resources as needed and/or on whatever came up rather than in 
pursuit of strategic goals. 


A second weakness was having financial resources that were wholly inadequate. Evaluators 
consistently noted that when financial resources were not adequate to execute core 
authorizing activities, authorizers tended to cut back on/not execute ongoing monitoring 
responsibilities and academic oversight functions. 


Particularly when authorizing responsibilities were paid by per-student funding, evaluators 
noted the significant budgetary risk of very large schools. A large disincentive existed to 
close schools when that closure would have had a large impact on the stability of an 
authorizer’s budget. 


Leadership and Decision-Making Body 


Standard: The authorizer leadership and decision-making body understand their roles and 
responsibilities; are invested in the mission, vision, and strategic plan of authorizing; and 
have the expertise necessary to make well-informed decisions that support the tenets of a 
high-quality authorizer. 
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Key Strengths 


Mission-driven authorizer leadership was among the clearest documented strengths, in that 
leadership had a very clear understanding of the kind of authorizer they wanted to be and a 
clear description of the organizational decision-making structure they wanted to deploy 
aligned with their mission. Evaluators cited the following examples: 


e Frequent reference to the authorizer’s mission in decision making and the alignment 
of authorizing structures and systems with that mission 


e Periodic review—typically as a part of the strategic planning process—of the 
authorizer’s mission and purpose 


e Mission-aligned policies created by the board working with staff 


e The centering of the organization on its mission by the board and monitoring 
progress on outcomes, but not trying to manage other organizational functions 


e Expectation that staff would continuously develop expertise in leading or managing 
core authorizing functions. 


Leadership’s vocal commitment to being a rigorous authorizer was also noted as a strength. 
There was a level of pride and cultural commitment among senior leadership and board 
members in being a quality authorizer. Leadership described such things as (a) touting the 
rigor of their authorizing processes, (b) wanting to be known not as anti-charter but rather as 
pro-quality and being unflinching in rejecting any application or closing a school not deemed 
of quality, and (c) holding firm in their high-stakes decisions even when faced with political 
pressure to approve or renew applications and schools they deemed not meeting their 
quality bar. 


Expertise areas among senior leadership and board members were wide and varied. The 
most prevalent expertise areas appeared to be K-12 education experience, but expertise in 
business, fundraising, finance, and organizational development were also frequently 
mentioned areas. 


Key Weaknesses 


The most consistently mentioned weakness in leadership and decision making was the lack 
of onboarding, training, or development processes for authorizing board members. Board 
members appeared to be selected (or elected) without a formal process of orienting them to 
the board, and learning about authorizing and the authorizer’s systems was ad hoc and 
done exclusively by participating in high-stakes decision making. As evaluators noted, a lack 
of development for board members resulted in board members not having a full grasp of 
what authorizers do and subsequently advocating for things outside of an authorizer’s 
purview (e.g., wanting to provide resources and inappropriate technical assistance to 
schools or advocating for schools to adopt board member personal perspectives of effective 
schooling). 
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A lack of policies governing leadership and decision making or lack of documentation of 
such policies was also a weakness. Evaluators noted that a lack of formal processes and 
procedures for governance and decision making resulted in questions about the degree to 
which decisions were based on merit versus “politics” or emotional pleas from school 
officials. 
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Appendix A: Authorizer Evaluation Coded Domains, 
Sub-Domains, and Standards 


Application Decision Making 


Application Materials and Processes: The authorizer provides clear guidance and 
requirements regarding application materials and submission requirements and runs 
a clear and well-structured application process with realistic timelines. 

Educational Program: The authorizer has thorough requirements and rigorous 
evaluation criteria for the proposed educational program, including vision and 
mission statements, educational philosophy, curriculum and instruction, teaching 
skills and experience, calendar and daily schedule, target population, enrollment, 
and plans for educating students with special needs. 

Organizational Plan: The authorizer has thorough requirements and rigorous 
evaluation criteria for the proposed organizational plan including effective 
governance and management structures and systems (including staffing); the 
demonstration by founding team members of diverse and necessary capabilities; and 
the understanding of legal requirements related to opening and operating a charter 
school. 

Business and Financial Plan: The authorizer has thorough requirements and rigorous 
evaluation criteria for the proposed business plan, including financial viability of the 
plan demonstrated through budget projections that are aligned with the proposed 
educational program. 

Capacity: The authorizer has thorough requirements and rigorous criteria for 
evaluating the applicant’s capacity to implement the school plan effectively, including 
but not limited to a substantive in-person capacity interview with all qualified 
applicants. 

Priorities and Application Adaptation: The authorizer’s application includes 
requirements that specialized applicant types (i.e., charter network applicants, virtual 
school applicants, or applicants planning to contract with an educational service 
provider) provide additional relevant information, and to the extent applicable, the 
authorizer adapts its application to address identified needs or attract desired 
program types. 

Decision Alignment: The authorizer makes application decisions that are informed by 
and align with documented evidence and analysis of the extent to which the plan 
satisfies approval criteria and the extent to which applicants demonstrate strong 
preparation and capacity to establish and operate a quality charter school. 
Transparency: The authorizer has transparent processes for both application 
evaluation and application decision making. 


Performance-Based Accountability 


Educational Performance: The authorizer holds schools accountable for academic 
performance using objective and verifiable measures established in the charter 
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contract or performance framework that address, at a minimum, student 
achievement, student growth, and post-secondary success as the primary measures 
of school quality. 

Financial Performance: The authorizer holds schools accountable for financial 
performance using appropriate near-term and sustainability measures established in 
the charter contract or performance framework as the primary indicators of a 
school’s financial viability. 

Organizational Performance: The authorizer holds schools accountable for 
compliance with organizational performance requirements established in the charter 
contract or the performance framework, including educational program 
requirements, governance and reporting, financial management and oversight, and 
operational requirements related to students, employees, and the school 
environment. 

Decision Alignment: The authorizer makes accountability decisions that are informed 
by and align with documented evidence and analysis of the extent to which the 
school satisfies performance expectations. The analysis presented to decision 
makers is of high quality and the merits of the decisions themselves show decision 
making is based on thoughtful analysis ensuring that only the charter schools that 
meet or exceed expectations are in operation. 


Performance Management Systems 


Contracting: The authorizer executes a charter contract for each school that clearly 
articulates the rights and responsibilities of each party. 

School Openings: The authorizer ensures that approved schools are prepared 
adequately for opening. 

Ongoing Monitoring: The authorizer has an effective process for monitoring 
education, financial, and organizational performance of the schools it authorizes. 
School Intervention/Revocation: The authorizer has effective policies and practices 
for school intervention and revocation and conducts merit-based interventions, 
including revocation where appropriate, in response to clearly identified deficiencies 
in the school’s record of educational, organizational, and/or financial performance. 
Renewal: The authorizer runs a well-structured renewal process, including clear 
requirements, a meaningful opportunity for the school to present information and 
respond to the authorizer’s findings, clear communication, and prompt notification of 
decisions. 

Closure: Following non-renewal, revocation, or voluntary return of the charter, the 
authorizer has an effective plan for and ensures the orderly closure of a school. 
Transparency: The authorizer communicates to schools and the public clearly and 
consistently regarding expectations for and status of school performance, including 
formal reporting on school performance at least annually. 
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Autonomy 


Autonomy: The authorizer defines and respects the autonomies to which the schools 
are entitled based on statute, waiver, or authorizer policy. The authorizer does not 
reduce school autonomy unless there is a compelling reason to do so. 

Educational Program: The authorizer defines and respects school autonomy over the 
educational program. 

Financial Management: The authorizer defines and respects school autonomy over 
financial operations. 

Differentiated Oversight: The authorizer periodically reviews compliance 
requirements and evaluates the potential to differentiate school oversight based on 
flexibility in the law, demonstrated school performance, and other considerations. 


Organizational Capacity 


Strategic Plan: The authorizer plans well for the future in a way that aligns with 
NACSA’s Principles & Standards. The authorizer uses quality authorizing to forward 
its mission. 

Organizational Structure: The authorizer purposefully and economically staffs its 
office to effectively carry out its authorizing duties. Staff positions are clearly defined 
both in policy and in practice. 

Human Capital Processes and Systems: The authorizer has systems necessary for 
building and maintaining a strong workforce and implements them with fidelity. 
Conflicts of Interest: The authorizer operates free from conflicts of interest. 
Organizational Budget: The authorizer’s budget allows for organizational 
effectiveness and stability. The budget is aligned with the strategic goals and 
supports quality authorizing practice. 

Leadership and Decision-Making Body: The authorizer leadership and decision- 
making body understand their roles and responsibilities; are invested in the mission, 
vision, and strategic plan of authorizing; and have the expertise necessary to make 
well-informed decisions that support the tenets of a high-quality authorizer. 
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Appendix B: Comparison of Authorizers Comprising 
the Sample Compared to All Large Authorizers and 
All Authorizers 


Sample ae angen All Authorizers 

Size 

Large 89% N/A 11% 

Small 11% N/A 89% 
Type 

LEA 47% 52% 90% 

SEA 11% 15% 2% 

ICB 17% 9% 2% 

NFP 22% 9% 2% 

NEG 3% 2% 0.5% 

HEI 0% 12% A% 
Geography 

West 25% 29% 36% 

Midwest 39% 29% 20% 

South 25% 28% 31% 

East 11% 14% 12% 


Note: Percentages may not equal 100 due to rounding. Data for large authorizers and all authorizers 
are from the 2014-2015 year. Sample includes 46 independent evaluations. “Large” means the 
authorizer had 10 or more charter schools in their portfolio. “Small” means authorizer had 9 or fewer 
charter schools in their portfolio. LEA = Local education agency; SEA = State education agency; ICB = 
Independent charter board; NFP = Not for profit organization; NEG = Non-educational governmental 
organization; HEI = Higher education institution. 
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